Bluesky crash: psutil.NoSuchProcess when sending filename to Rigaku #231

qzhang234 · 2020-09-29T15:28:49Z

Error message from terminal


X020_02637_att00_Test
HDF5 workflow file name: /home/8ididata/2020-2/bluesky202009/X020_02637_att00_Test/X020_02637_att00_Test_0001-100000.hdf      
X020_02638_att00_Test
HDF5 workflow file name: /home/8ididata/2020-2/bluesky202009/X020_02638_att00_Test/X020_02638_att00_Test_0001-100000.hdf      
X020_02639_att00_Test
---------------------------------------------------------------------------
ProcessLookupError                        Traceback (most recent call last)
~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1515         try:
-> 1516             return fun(self, *args, **kwargs)
   1517         except PermissionError:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in threads(self)
   1903                 with open_binary(fname) as f:
-> 1904                     st = f.read().strip()
   1905             except FileNotFoundError:

ProcessLookupError: [Errno 3] No such process

During handling of the above exception, another exception occurred:

NoSuchProcess                             Traceback (most recent call last)
<ipython-input-7-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-2/bp_Test_Bluesky.py in multi_capil()
     95     z0 = 0
     96 
---> 97     yield from repeat_acq(sample_name='Test',att_value=0,acq_rep=10000,samx_cen=0,samz_cen=0)
     98 
     99     # yield from repeat_acq(sample_name='D100_NoFly',att_value=2,acq_rep=5,samx_cen=9.4,samz_cen=z0)

/home/beams10/8IDIUSER/bluesky_data/2020/2020-2/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    316 
    317     logger.info("calling full_acquire_procedure()")
--> 318     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    284         logger.debug("before count()")
    285         # yield from bp.count([areadet], md=md)
--> 286         yield from inner_count([areadet], md=_md)
    287         logger.debug("after count()")
    288 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    237         yield from bps.open_run(md=md)
    238         for obj in devices:
--> 239             yield from bps.stage(obj)
    240         grp = bps._short_uid('trigger')
    241         no_wait = True

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in stage(obj)
    694     :func:`bluesky.plan_stubs.unstage`
    695     """
--> 696     return (yield Msg('stage', obj))
    697 
    698 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _stage(self, msg)
   2166         if not hasattr(obj, 'stage'):
   2167             return []
-> 2168         result = obj.stage()
   2169         self._staged.add(obj)  # add first in case of failure below
   2170         await self._reset_checkpoint_state_coro()

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in stage(self)
    220         shutter_control.put("Open")
    221         cmd = f"echo FILE:F:{self.batch_name.get()} | nc rigaku1.xray.aps.anl.gov 10000"
--> 222         self.unix_process.put(cmd)
    223 
    224     def trigger(self):

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in put(self, unix_command)
     89 
     90     def put(self, unix_command):
---> 91         status = self.set(unix_command)
     92         while not status.done:
     93             time.sleep(0.01)

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in set(self, unix_command)
     60 
     61         # see: https://github.com/aps-8id-dys/ipython-8idiuser/issues/195
---> 62         pi = get_process_info()
     63         logger.info(
     64             "%s [#open files=%d, #open fd=%d, #threads=%d, #connections=%d, #context switches=%d]",

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info()
     33 
     34 def get_process_info():
---> 35     pd = process_info.as_dict()
     36     return dict(
     37         memory_info=pd["memory_info"],

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in as_dict(self, attrs, ad_value)
    536                     else:
    537                         meth = getattr(self, name)
--> 538                         ret = meth()
    539                 except (AccessDenied, ZombieProcess):
    540                     ret = ad_value

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in threads(self)
    884             On OpenBSD this method requires root access.
    885             """
--> 886             return self._proc.threads()
    887 
    888     @_assert_pid_not_reused

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1518             raise AccessDenied(self.pid, self._name)
   1519         except ProcessLookupError:
-> 1520             raise NoSuchProcess(self.pid, self._name)
   1521         except FileNotFoundError:
   1522             if not os.path.exists("%s/%s" % (self._procfs_path, self.pid)):

NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=30216, name='ipython')

In [8]:

The text was updated successfully, but these errors were encountered:

prjemian · 2020-09-29T16:18:55Z

Youch! Process no longer exists?

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in stage(self)
    220         shutter_control.put("Open")
    221         cmd = f"echo FILE:F:{self.batch_name.get()} | nc rigaku1.xray.aps.anl.gov 10000"
--> 222         self.unix_process.put(cmd)
    223 
    224     def trigger(self):

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in put(self, unix_command)
     89 
     90     def put(self, unix_command):
---> 91         status = self.set(unix_command)
     92         while not status.done:
     93             time.sleep(0.01)

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in set(self, unix_command)
     60 
     61         # see: https://github.com/aps-8id-dys/ipython-8idiuser/issues/195
---> 62         pi = get_process_info()
     63         logger.info(
     64             "%s [#open files=%d, #open fd=%d, #threads=%d, #connections=%d, #context switches=%d]",

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info()
     33 
     34 def get_process_info():
---> 35     pd = process_info.as_dict()
     36     return dict(
     37         memory_info=pd["memory_info"],

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in as_dict(self, attrs, ad_value)
    536                     else:
    537                         meth = getattr(self, name)
--> 538                         ret = meth()
    539                 except (AccessDenied, ZombieProcess):
    540                     ret = ad_value

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in threads(self)
    884             On OpenBSD this method requires root access.
    885             """
--> 886             return self._proc.threads()
    887 
    888     @_assert_pid_not_reused

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1518             raise AccessDenied(self.pid, self._name)
   1519         except ProcessLookupError:
-> 1520             raise NoSuchProcess(self.pid, self._name)
   1521         except FileNotFoundError:
   1522             if not os.path.exists("%s/%s" % (self._procfs_path, self.pid)):

NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=30216, name='ipython')

We have set something to None that we should not have?

prjemian · 2020-09-29T16:20:08Z

Is this different from #230?

qzhang234 · 2020-09-29T17:06:59Z

I do not think so, for the Rigaku detector is in 'Ready' state this time and it does not show the 'Hang' state as in #230

prjemian · 2020-09-29T17:14:02Z

We have a sleep time of 10 ms while waiting for Rigaku .set(unix_command) in def put(). Maybe decrease that to 1 ms to reduce chance of missing a quick toggle by Rigaku's LabView software.

prjemian · 2020-09-29T17:15:59Z

The failure was in Python as the new file name was to be sent to the detector. The failure happened because the psutils package could not find the process that sends the unix command. What happened to it?

qzhang234 · 2020-09-29T17:34:35Z

Excerpt from /home/beams/8IDIUSER/bluesky_data/2020/2020-2/.logs/ipython_logger.log

Last lines from logger


|2020-09-28 22:28:55.366|INFO|30216|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - X020_02637_att00_Test
|2020-09-28 22:28:55.367|INFO|30216|bluesky-session|xpcs_acquire|48|bluesky-run-engine| - AD_Acquire starting
|2020-09-28 22:28:55.367|INFO|30216|bluesky-session|xpcs_acquire|59|bluesky-run-engine| - file_path = /home/8ididata/2020-2/bluesky202009/X020_02637_att00_Test/
|2020-09-28 22:28:55.392|INFO|30216|bluesky-session|xpcs_acquire|317|bluesky-run-engine| - calling full_acquire_procedure()
|2020-09-28 22:28:55.401|INFO|30216|bluesky-session|xpcs_acquire|146|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-09-28 22:28:55.473|INFO|30216|bluesky-session|xpcs_acquire|281|bluesky-run-engine| - scaler should be autocounting now
|2020-09-28 22:28:55.558|INFO|30216|bluesky-session|rigaku_ufxc|117|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-09-28 22:28:55.591|INFO|30216|bluesky-session|xpcs_acquire|312|Thread-235338| - DM workflow kickoff done
|2020-09-28 22:28:55.591|INFO|30216|bluesky-session|xpcs_acquire|313|Thread-235338| - b'id=eced8539-48d6-44d7-8755-b8d29ee72143 owner=8idiuser status=pending startTime=1601350135.57 startTimestamp=2020/09/28 22:28:55 CDT \n'
|2020-09-28 22:28:55.630|INFO|30216|bluesky-session|rigaku_ufxc|63|bluesky-run-engine| - echo FILE:F:X020_02637_att00_Test | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=43, #threads=41, #connections=11, #context switches=2]
|2020-09-28 22:28:56.105|INFO|30216|bluesky-session|rigaku_ufxc|63|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-09-28 22:28:59.800|INFO|30216|bluesky-session|xpcs_acquire|307|Thread-235427| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-2/bluesky202009/X020_02637_att00_Test/X020_02637_att00_Test_0001-100000.hdf
|2020-09-28 22:28:59.831|INFO|30216|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - X020_02638_att00_Test
|2020-09-28 22:28:59.832|INFO|30216|bluesky-session|xpcs_acquire|48|bluesky-run-engine| - AD_Acquire starting
|2020-09-28 22:28:59.832|INFO|30216|bluesky-session|xpcs_acquire|59|bluesky-run-engine| - file_path = /home/8ididata/2020-2/bluesky202009/X020_02638_att00_Test/
|2020-09-28 22:28:59.884|INFO|30216|bluesky-session|xpcs_acquire|317|bluesky-run-engine| - calling full_acquire_procedure()
|2020-09-28 22:28:59.894|INFO|30216|bluesky-session|xpcs_acquire|146|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-09-28 22:28:59.958|INFO|30216|bluesky-session|xpcs_acquire|281|bluesky-run-engine| - scaler should be autocounting now
|2020-09-28 22:29:00.016|INFO|30216|bluesky-session|rigaku_ufxc|117|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-09-28 22:29:00.038|INFO|30216|bluesky-session|xpcs_acquire|312|Thread-235427| - DM workflow kickoff done
|2020-09-28 22:29:00.042|INFO|30216|bluesky-session|xpcs_acquire|313|Thread-235427| - b'id=ebb8e8ba-05a8-419b-b8a2-df18823ef064 owner=8idiuser status=pending startTime=1601350140.02 startTimestamp=2020/09/28 22:29:00 CDT \n'
|2020-09-28 22:29:00.087|INFO|30216|bluesky-session|rigaku_ufxc|63|bluesky-run-engine| - echo FILE:F:X020_02638_att00_Test | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=41, #connections=11, #context switches=2]
|2020-09-28 22:29:00.543|INFO|30216|bluesky-session|rigaku_ufxc|63|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-09-28 22:29:04.094|INFO|30216|bluesky-session|xpcs_acquire|307|Thread-235516| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-2/bluesky202009/X020_02638_att00_Test/X020_02638_att00_Test_0001-100000.hdf
|2020-09-28 22:29:04.163|INFO|30216|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - X020_02639_att00_Test
|2020-09-28 22:29:04.164|INFO|30216|bluesky-session|xpcs_acquire|48|bluesky-run-engine| - AD_Acquire starting
|2020-09-28 22:29:04.164|INFO|30216|bluesky-session|xpcs_acquire|59|bluesky-run-engine| - file_path = /home/8ididata/2020-2/bluesky202009/X020_02639_att00_Test/
|2020-09-28 22:29:04.186|INFO|30216|bluesky-session|xpcs_acquire|317|bluesky-run-engine| - calling full_acquire_procedure()
|2020-09-28 22:29:04.195|INFO|30216|bluesky-session|xpcs_acquire|146|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-09-28 22:29:04.260|INFO|30216|bluesky-session|xpcs_acquire|281|bluesky-run-engine| - scaler should be autocounting now
|2020-09-28 22:29:04.316|INFO|30216|bluesky-session|rigaku_ufxc|117|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-09-28 22:29:04.346|INFO|30216|bluesky-session|xpcs_acquire|312|Thread-235516| - DM workflow kickoff done
|2020-09-28 22:29:04.346|INFO|30216|bluesky-session|xpcs_acquire|313|Thread-235516| - b'id=f9c177f2-d856-4346-9a08-a32f690102e2 owner=8idiuser status=pending startTime=1601350144.33 startTimestamp=2020/09/28 22:29:04 CDT \n'

qzhang234 · 2020-09-29T17:35:29Z

Also screenshot from DM Workflow analysis web GUI:

qzhang234 · 2020-09-29T17:37:06Z

@prjemian If you look at the logger file, it seems that the file name for 2639 did get sent, and it even started the DM workflow, however for some reason the ipython terminal crashed at the 'sending file name' part.

Any suggestions?

prjemian · 2020-09-29T17:38:43Z

Technically, the logger may report it but python failed to find the process to send the unix command to the detector. Probably logged just before sending the command.

qzhang234 · 2020-09-29T19:01:08Z

@prjemian Bluesky crashed again, this time at only 192 measurements:

Screen output


HDF5 workflow file name: /home/8ididata/2020-2/bluesky202009/X021_00190_att00_Test/X021_00190_att00_Test_0001-100000.hdf      
X021_00191_att00_Test
HDF5 workflow file name: /home/8ididata/2020-2/bluesky202009/X021_00191_att00_Test/X021_00191_att00_Test_0001-100000.hdf      
X021_00192_att00_Test
HDF5 workflow file name: /home/8ididata/2020-2/bluesky202009/X021_00192_att00_Test/X021_00192_att00_Test_0001-100000.hdf      
X021_00193_att00_Test
---------------------------------------------------------------------------
ProcessLookupError                        Traceback (most recent call last)
~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1515         try:
-> 1516             return fun(self, *args, **kwargs)
   1517         except PermissionError:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in threads(self)
   1903                 with open_binary(fname) as f:
-> 1904                     st = f.read().strip()
   1905             except FileNotFoundError:

ProcessLookupError: [Errno 3] No such process

During handling of the above exception, another exception occurred:

NoSuchProcess                             Traceback (most recent call last)
<ipython-input-2-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-2/bp_Test_Bluesky.py in multi_capil()
     96     z0 = 0
     97 
---> 98     yield from repeat_acq(sample_name='Test',att_value=0,acq_rep=10000,samx_cen=0,samz_cen=0)
     99 
    100     # yield from repeat_acq(sample_name='D100_NoFly',att_value=2,acq_rep=5,samx_cen=9.4,samz_cen=z0)

/home/beams10/8IDIUSER/bluesky_data/2020/2020-2/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    238         yield from bps.open_run(md=md)
    239         for obj in devices:
--> 240             yield from bps.stage(obj)
    241         grp = bps._short_uid('trigger')
    242         no_wait = True

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in stage(obj)
    694     :func:`bluesky.plan_stubs.unstage`
    695     """
--> 696     return (yield Msg('stage', obj))
    697 
    698 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _stage(self, msg)
   2166         if not hasattr(obj, 'stage'):
   2167             return []
-> 2168         result = obj.stage()
   2169         self._staged.add(obj)  # add first in case of failure below
   2170         await self._reset_checkpoint_state_coro()

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in stage(self)
    220         shutter_control.put("Open")
    221         cmd = f"echo FILE:F:{self.batch_name.get()} | nc rigaku1.xray.aps.anl.gov 10000"
--> 222         self.unix_process.put(cmd)
    223 
    224     def trigger(self):

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in put(self, unix_command)
     89 
     90     def put(self, unix_command):
---> 91         status = self.set(unix_command)
     92         while not status.done:
     93             time.sleep(0.001)

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in set(self, unix_command)
     60 
     61         # see: https://github.com/aps-8id-dys/ipython-8idiuser/issues/195
---> 62         pi = get_process_info()
     63         logger.info(
     64             "%s [#open files=%d, #open fd=%d, #threads=%d, #connections=%d, #context switches=%d]",

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info()
     33 
     34 def get_process_info():
---> 35     pd = process_info.as_dict()
     36     return dict(
     37         memory_info=pd["memory_info"],

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in as_dict(self, attrs, ad_value)
    536                     else:
    537                         meth = getattr(self, name)
--> 538                         ret = meth()
    539                 except (AccessDenied, ZombieProcess):
    540                     ret = ad_value

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in threads(self)
    884             On OpenBSD this method requires root access.
    885             """
--> 886             return self._proc.threads()
    887 
    888     @_assert_pid_not_reused

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1518             raise AccessDenied(self.pid, self._name)
   1519         except ProcessLookupError:
-> 1520             raise NoSuchProcess(self.pid, self._name)
   1521         except FileNotFoundError:
   1522             if not os.path.exists("%s/%s" % (self._procfs_path, self.pid)):

NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=8074, name='ipython')

In [3]:

qzhang234 · 2020-09-29T20:16:33Z

Last few lines of the logger file:

/2020-2/ipython_logger.log


|2020-09-29 12:52:09.672|INFO|8074|bluesky-session|xpcs_acquire|308|Thread-17042| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-2/bluesky202009/X021_00190_att00_Test/X021_00190_att00_Test_0001-100000.hdf
|2020-09-29 12:52:09.686|INFO|8074|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - X021_00191_att00_Test
|2020-09-29 12:52:09.687|INFO|8074|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-09-29 12:52:09.687|INFO|8074|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-2/bluesky202009/X021_00191_att00_Test/
|2020-09-29 12:52:09.709|INFO|8074|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-09-29 12:52:09.718|INFO|8074|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-09-29 12:52:09.781|INFO|8074|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-09-29 12:52:09.870|INFO|8074|bluesky-session|rigaku_ufxc|117|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-09-29 12:52:09.916|INFO|8074|bluesky-session|xpcs_acquire|313|Thread-17042| - DM workflow kickoff done
|2020-09-29 12:52:09.916|INFO|8074|bluesky-session|xpcs_acquire|314|Thread-17042| - b'id=e404b543-03f2-4bb2-a095-0d79d266a280 owner=8idiuser status=pending startTime=1601401929.9 startTimestamp=2020/09/29 12:52:09 CDT \n'
|2020-09-29 12:52:09.919|INFO|8074|bluesky-session|rigaku_ufxc|63|bluesky-run-engine| - echo FILE:F:X021_00191_att00_Test | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=43, #threads=43, #connections=11, #context switches=2]
|2020-09-29 12:52:10.325|INFO|8074|bluesky-session|rigaku_ufxc|63|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-09-29 12:52:13.738|INFO|8074|bluesky-session|xpcs_acquire|308|Thread-17131| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-2/bluesky202009/X021_00191_att00_Test/X021_00191_att00_Test_0001-100000.hdf
|2020-09-29 12:52:13.775|INFO|8074|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - X021_00192_att00_Test
|2020-09-29 12:52:13.776|INFO|8074|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-09-29 12:52:13.776|INFO|8074|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-2/bluesky202009/X021_00192_att00_Test/
|2020-09-29 12:52:13.799|INFO|8074|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-09-29 12:52:13.813|INFO|8074|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-09-29 12:52:13.880|INFO|8074|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-09-29 12:52:13.933|INFO|8074|bluesky-session|rigaku_ufxc|117|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-09-29 12:52:13.936|INFO|8074|bluesky-session|xpcs_acquire|313|Thread-17131| - DM workflow kickoff done
|2020-09-29 12:52:13.936|INFO|8074|bluesky-session|xpcs_acquire|314|Thread-17131| - b'id=f574430c-dbbd-4e10-96b3-1a05489d9b61 owner=8idiuser status=pending startTime=1601401933.91 startTimestamp=2020/09/29 12:52:13 CDT \n'
|2020-09-29 12:52:13.979|INFO|8074|bluesky-session|rigaku_ufxc|63|bluesky-run-engine| - echo FILE:F:X021_00192_att00_Test | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-09-29 12:52:14.395|INFO|8074|bluesky-session|rigaku_ufxc|63|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-09-29 12:52:17.977|INFO|8074|bluesky-session|xpcs_acquire|308|Thread-17220| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-2/bluesky202009/X021_00192_att00_Test/X021_00192_att00_Test_0001-100000.hdf
|2020-09-29 12:52:17.998|INFO|8074|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - X021_00193_att00_Test
|2020-09-29 12:52:18.000|INFO|8074|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-09-29 12:52:18.000|INFO|8074|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-2/bluesky202009/X021_00193_att00_Test/
|2020-09-29 12:52:18.057|INFO|8074|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-09-29 12:52:18.067|INFO|8074|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-09-29 12:52:18.103|INFO|8074|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-09-29 12:52:18.201|INFO|8074|bluesky-session|xpcs_acquire|313|Thread-17220| - DM workflow kickoff done
|2020-09-29 12:52:18.201|INFO|8074|bluesky-session|xpcs_acquire|314|Thread-17220| - b'id=b99eb360-8dbb-412b-a878-eb82b67a7fcb owner=8idiuser status=pending startTime=1601401938.18 startTimestamp=2020/09/29 12:52:18 CDT \n'
|2020-09-29 12:52:18.204|INFO|8074|bluesky-session|rigaku_ufxc|117|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.


</details>

qzhang234 · 2020-09-29T20:19:49Z

Screenshot from LabView. The detector is in Ready state as well.

qzhang234 · 2020-09-29T20:24:01Z

From my untrained eye this crash looks exactly the same as the previous one (psutil.NoSuchProcess process no longer exists), except the pid is different.

@prjemian Please let me know when this problem is fixed so that I can start the Bluesky testing again.

prjemian · 2020-10-01T21:21:44Z

https://psutil.readthedocs.io/en/latest/#psutil.NoSuchProcess

qzhang234 · 2020-10-02T15:36:39Z

@prjemian C026 crashed at measurement 10165 at 10/02, 10:01 am (ran for ~ 13 hours). Same error:

Error from Ipython terminal

C026_CPMV_att00_10163
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10163/C026_CPMV_att00_10163_0001-100000.hdf  
C026_CPMV_att00_10164
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10164/C026_CPMV_att00_10164_0001-100000.hdf  
C026_CPMV_att00_10165                                                                                                    
---------------------------------------------------------------------------                                              
ProcessLookupError                        Traceback (most recent call last)
~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1515         try:
-> 1516             return fun(self, *args, **kwargs)
   1517         except PermissionError:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in threads(self)
   1903                 with open_binary(fname) as f:
-> 1904                     st = f.read().strip()
   1905             except FileNotFoundError:

ProcessLookupError: [Errno 3] No such process

During handling of the above exception, another exception occurred:

NoSuchProcess                             Traceback (most recent call last)
~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info()
     36     try:
---> 37         pd = process_info.as_dict()
     38     except psutil.NoSuchProcess as exc:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in as_dict(self, attrs, ad_value)
    537                         meth = getattr(self, name)
--> 538                         ret = meth()
    539                 except (AccessDenied, ZombieProcess):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in threads(self)
    885             """
--> 886             return self._proc.threads()
    887 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1519         except ProcessLookupError:
-> 1520             raise NoSuchProcess(self.pid, self._name)
   1521         except FileNotFoundError:

NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=20292, name='ipython')

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 0.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV',att_value=0,acq_rep=50000,samx_cen=2.2,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    238         yield from bps.open_run(md=md)
    239         for obj in devices:
--> 240             yield from bps.stage(obj)
    241         grp = bps._short_uid('trigger')
    242         no_wait = True

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in stage(obj)
    694     :func:`bluesky.plan_stubs.unstage`
    695     """
--> 696     return (yield Msg('stage', obj))
    697 
    698 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _stage(self, msg)
   2166         if not hasattr(obj, 'stage'):
   2167             return []
-> 2168         result = obj.stage()
   2169         self._staged.add(obj)  # add first in case of failure below
   2170         await self._reset_checkpoint_state_coro()

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in stage(self)
    225         shutter_control.put("Open")
    226         cmd = f"echo FILE:F:{self.batch_name.get()} | nc rigaku1.xray.aps.anl.gov 10000"
--> 227         self.unix_process.put(cmd)
    228 
    229     def trigger(self):

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in put(self, unix_command)
     94 
     95     def put(self, unix_command):
---> 96         status = self.set(unix_command)
     97         while not status.done:
     98             time.sleep(0.001)

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in set(self, unix_command)
     65 
     66         # see: https://github.com/aps-8id-dys/ipython-8idiuser/issues/195
---> 67         pi = get_process_info()
     68         logger.info(
     69             "%s [#open files=%d, #open fd=%d, #threads=%d, #connections=%d, #context switches=%d]",

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info()
     38     except psutil.NoSuchProcess as exc:
     39         logger.error("process_info status: %s", process_info.status())
---> 40         raise RuntimeError(f"ProcessNotFound: {exc}")
     41     return dict(
     42         memory_info=pd["memory_info"],

RuntimeError: ProcessNotFound: psutil.NoSuchProcess process no longer exists (pid=20292, name='ipython')

In [9]:

Last lines from ipython_logger.log. Only one occurrence of ERROR, at the end of the file:

Logger


|2020-10-02 10:01:40.923|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10156
|2020-10-02 10:01:40.923|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:01:40.923|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10156/
|2020-10-02 10:01:40.969|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:01:40.979|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:01:41.019|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:01:41.020|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-936522| - DM workflow kickoff done
|2020-10-02 10:01:41.020|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-936522| - b'id=483ab1f1-abc7-42b7-a6a0-f3b9b6c9428c owner=8idiuser status=pending startTime=1601650901.0 startTimestamp=2020/10/02 10:01:41 CDT \n'
|2020-10-02 10:01:41.078|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:01:41.079|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:01:41.197|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10156 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=42, #connections=11, #context switches=2]
|2020-10-02 10:01:41.680|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:01:41.817|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:01:45.601|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-936614| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10156/C026_CPMV_att00_10156_0001-100000.hdf
|2020-10-02 10:01:45.867|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10157
|2020-10-02 10:01:45.868|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:01:45.868|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10157/
|2020-10-02 10:01:45.889|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:01:45.899|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:01:45.937|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-936614| - DM workflow kickoff done
|2020-10-02 10:01:45.937|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-936614| - b'id=d4777f9c-4f47-4ecb-bb34-81724934ccf5 owner=8idiuser status=pending startTime=1601650905.92 startTimestamp=2020/10/02 10:01:45 CDT \n'
|2020-10-02 10:01:45.968|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:01:46.060|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:01:46.061|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:01:46.171|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10157 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=42, #connections=11, #context switches=2]
|2020-10-02 10:01:46.655|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:01:46.764|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:01:50.318|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-936706| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10157/C026_CPMV_att00_10157_0001-100000.hdf
|2020-10-02 10:01:50.573|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10158
|2020-10-02 10:01:50.573|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:01:50.573|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10158/
|2020-10-02 10:01:50.593|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:01:50.603|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:01:50.648|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-936706| - DM workflow kickoff done
|2020-10-02 10:01:50.649|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-936706| - b'id=c0ff66f2-e8b3-499f-9161-c7d4343ab07a owner=8idiuser status=pending startTime=1601650910.63 startTimestamp=2020/10/02 10:01:50 CDT \n'
|2020-10-02 10:01:50.668|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:01:50.727|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:01:50.727|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:01:50.837|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10158 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=42, #connections=11, #context switches=2]
|2020-10-02 10:01:51.223|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:01:51.329|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:01:55.033|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-936798| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10158/C026_CPMV_att00_10158_0001-100000.hdf
|2020-10-02 10:01:55.295|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10159
|2020-10-02 10:01:55.295|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:01:55.295|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10159/
|2020-10-02 10:01:55.317|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:01:55.330|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:01:55.379|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-936798| - DM workflow kickoff done
|2020-10-02 10:01:55.379|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-936798| - b'id=4776f8d9-6f51-4b6c-81c1-5aea2ee96f4d owner=8idiuser status=pending startTime=1601650915.36 startTimestamp=2020/10/02 10:01:55 CDT \n'
|2020-10-02 10:01:55.399|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:01:55.460|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:01:55.461|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:01:55.569|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10159 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=42, #connections=11, #context switches=2]
|2020-10-02 10:01:56.152|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:01:56.258|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:01:59.898|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-936890| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10159/C026_CPMV_att00_10159_0001-100000.hdf
|2020-10-02 10:02:00.167|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10160
|2020-10-02 10:02:00.167|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:02:00.167|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10160/
|2020-10-02 10:02:00.189|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:02:00.199|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:02:00.246|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-936890| - DM workflow kickoff done
|2020-10-02 10:02:00.247|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-936890| - b'id=88c8ab08-9ef6-4ad4-87b4-22a68c41e788 owner=8idiuser status=pending startTime=1601650920.23 startTimestamp=2020/10/02 10:02:00 CDT \n'
|2020-10-02 10:02:00.266|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:02:00.403|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:02:00.404|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:00.527|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10160 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=40, #connections=11, #context switches=2]
|2020-10-02 10:02:01.024|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:01.151|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:02:04.768|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-936982| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10160/C026_CPMV_att00_10160_0001-100000.hdf
|2020-10-02 10:02:05.025|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10161
|2020-10-02 10:02:05.025|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:02:05.025|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10161/
|2020-10-02 10:02:05.076|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:02:05.092|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:02:05.108|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-936982| - DM workflow kickoff done
|2020-10-02 10:02:05.108|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-936982| - b'id=012fc0cf-897c-494c-9900-bb681c118565 owner=8idiuser status=pending startTime=1601650925.09 startTimestamp=2020/10/02 10:02:05 CDT \n'
|2020-10-02 10:02:05.165|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:02:05.228|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:02:05.228|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:05.339|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10161 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=41, #connections=11, #context switches=2]
|2020-10-02 10:02:05.721|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:05.841|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:02:09.394|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-937074| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10161/C026_CPMV_att00_10161_0001-100000.hdf
|2020-10-02 10:02:09.623|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10162
|2020-10-02 10:02:09.623|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:02:09.624|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10162/
|2020-10-02 10:02:09.671|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:02:09.682|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:02:09.719|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:02:09.729|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-937074| - DM workflow kickoff done
|2020-10-02 10:02:09.729|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-937074| - b'id=472c2d61-b91b-4335-b0d8-93b7e1c7000d owner=8idiuser status=pending startTime=1601650929.71 startTimestamp=2020/10/02 10:02:09 CDT \n'
|2020-10-02 10:02:09.778|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:02:09.778|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:09.886|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10162 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=42, #connections=11, #context switches=2]
|2020-10-02 10:02:10.272|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:10.393|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:02:13.904|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-937166| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10162/C026_CPMV_att00_10162_0001-100000.hdf
|2020-10-02 10:02:14.165|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10163
|2020-10-02 10:02:14.165|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:02:14.165|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10163/
|2020-10-02 10:02:14.188|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:02:14.197|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:02:14.261|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-937166| - DM workflow kickoff done
|2020-10-02 10:02:14.261|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-937166| - b'id=966bd476-29d7-4b4c-b81d-8879f9bbce33 owner=8idiuser status=pending startTime=1601650934.24 startTimestamp=2020/10/02 10:02:14 CDT \n'
|2020-10-02 10:02:14.262|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:02:14.320|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:02:14.320|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:14.429|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10163 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=42, #connections=11, #context switches=2]
|2020-10-02 10:02:14.910|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:15.035|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:02:18.730|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-937258| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10163/C026_CPMV_att00_10163_0001-100000.hdf
|2020-10-02 10:02:18.991|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10164
|2020-10-02 10:02:18.991|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:02:18.991|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10164/
|2020-10-02 10:02:19.010|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:02:19.020|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:02:19.081|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-937258| - DM workflow kickoff done
|2020-10-02 10:02:19.081|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-937258| - b'id=75cb3f3b-71ca-43bb-8b4f-a752ac6da7ac owner=8idiuser status=pending startTime=1601650939.06 startTimestamp=2020/10/02 10:02:19 CDT \n'
|2020-10-02 10:02:19.088|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:02:19.161|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:02:19.162|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:19.271|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C026_CPMV_att00_10164 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=42, #connections=11, #context switches=2]
|2020-10-02 10:02:19.753|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:19.880|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=33, #connections=11, #context switches=2]
|2020-10-02 10:02:23.519|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-937350| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10164/C026_CPMV_att00_10164_0001-100000.hdf
|2020-10-02 10:02:23.845|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-937350| - DM workflow kickoff done
|2020-10-02 10:02:23.845|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-937350| - b'id=569f1237-4093-42ef-9cbc-d2357ea062be owner=8idiuser status=pending startTime=1601650943.83 startTimestamp=2020/10/02 10:02:23 CDT \n'
|2020-10-02 10:02:24.776|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C026_CPMV_att00_10165
|2020-10-02 10:02:24.776|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 10:02:24.776|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C026_CPMV_att00_10165/
|2020-10-02 10:02:24.803|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 10:02:24.815|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 10:02:24.885|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 10:02:24.976|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 10:02:24.976|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 10:02:24.977|ERROR|20292|bluesky-session|rigaku_ufxc|39|bluesky-run-engine| - process_info status: sleeping

LabView interface looks normal too:

Kacperswitalski · 2020-10-02T18:16:53Z

@prjemian C027 crashed at measurement 00085 at 1:10 pm on 10/02

Error message from terminal

C027_CPMV_att00_00073
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00073/C027_CPMV_att00_00073_0001-100000.hdf  
C027_CPMV_att00_00074
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00074/C027_CPMV_att00_00074_0001-100000.hdf  
C027_CPMV_att00_00075
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00075/C027_CPMV_att00_00075_0001-100000.hdf  
C027_CPMV_att00_00076
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00076/C027_CPMV_att00_00076_0001-100000.hdf  
C027_CPMV_att00_00077
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00077/C027_CPMV_att00_00077_0001-100000.hdf  
C027_CPMV_att00_00078
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00078/C027_CPMV_att00_00078_0001-100000.hdf  
C027_CPMV_att00_00079
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00079/C027_CPMV_att00_00079_0001-100000.hdf  
C027_CPMV_att00_00080
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00080/C027_CPMV_att00_00080_0001-100000.hdf  
C027_CPMV_att00_00081
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00081/C027_CPMV_att00_00081_0001-100000.hdf  
C027_CPMV_att00_00082
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00082/C027_CPMV_att00_00082_0001-100000.hdf  
C027_CPMV_att00_00083
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00083/C027_CPMV_att00_00083_0001-100000.hdf  
C027_CPMV_att00_00084
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00084/C027_CPMV_att00_00084_0001-100000.hdf  
C027_CPMV_att00_00085                                                                                                    
---------------------------------------------------------------------------                                              
ProcessLookupError                        Traceback (most recent call last)
~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1515         try:
-> 1516             return fun(self, *args, **kwargs)
   1517         except PermissionError:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in threads(self)
   1903                 with open_binary(fname) as f:
-> 1904                     st = f.read().strip()
   1905             except FileNotFoundError:

ProcessLookupError: [Errno 3] No such process

During handling of the above exception, another exception occurred:

NoSuchProcess                             Traceback (most recent call last)
~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info()
     36     try:
---> 37         pd = process_info.as_dict()
     38     except psutil.NoSuchProcess as exc:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in as_dict(self, attrs, ad_value)
    537                         meth = getattr(self, name)
--> 538                         ret = meth()
    539                 except (AccessDenied, ZombieProcess):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/__init__.py in threads(self)
    885             """
--> 886             return self._proc.threads()
    887 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/psutil/_pslinux.py in wrapper(self, *args, **kwargs)
   1519         except ProcessLookupError:
-> 1520             raise NoSuchProcess(self.pid, self._name)
   1521         except FileNotFoundError:

NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=20292, name='ipython')

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-15-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 0.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV',att_value=0,acq_rep=40000,samx_cen=2.2,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    238         yield from bps.open_run(md=md)
    239         for obj in devices:
--> 240             yield from bps.stage(obj)
    241         grp = bps._short_uid('trigger')
    242         no_wait = True

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in stage(obj)
    694     :func:`bluesky.plan_stubs.unstage`
    695     """
--> 696     return (yield Msg('stage', obj))
    697 
    698 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _stage(self, msg)
   2166         if not hasattr(obj, 'stage'):
   2167             return []
-> 2168         result = obj.stage()
   2169         self._staged.add(obj)  # add first in case of failure below
   2170         await self._reset_checkpoint_state_coro()

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in stage(self)
    225         shutter_control.put("Open")
    226         cmd = f"echo FILE:F:{self.batch_name.get()} | nc rigaku1.xray.aps.anl.gov 10000"
--> 227         self.unix_process.put(cmd)
    228 
    229     def trigger(self):

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in put(self, unix_command)
     94 
     95     def put(self, unix_command):
---> 96         status = self.set(unix_command)
     97         while not status.done:
     98             time.sleep(0.001)

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in set(self, unix_command)
     65 
     66         # see: https://github.com/aps-8id-dys/ipython-8idiuser/issues/195
---> 67         pi = get_process_info()
     68         logger.info(
     69             "%s [#open files=%d, #open fd=%d, #threads=%d, #connections=%d, #context switches=%d]",

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info()
     38     except psutil.NoSuchProcess as exc:
     39         logger.error("process_info status: %s", process_info.status())
---> 40         raise RuntimeError(f"ProcessNotFound: {exc}")
     41     return dict(
     42         memory_info=pd["memory_info"],

RuntimeError: ProcessNotFound: psutil.NoSuchProcess process no longer exists (pid=20292, name='ipython')

Logger


|2020-10-02 13:10:07.357|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-944641| - DM workflow kickoff done
|2020-10-02 13:10:07.357|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-944641| - b'id=cf0a24e9-f563-44a9-aff2-03694feee8ee owner=8idiuser status=pending startTime=1601662207.34 startTimestamp=2020/10/02 13:10:07 CDT \n'
|2020-10-02 13:10:07.391|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:07.392|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:07.493|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00075 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:07.981|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:08.100|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:11.619|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-944733| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00075/C027_CPMV_att00_00075_0001-100000.hdf
|2020-10-02 13:10:11.872|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00076
|2020-10-02 13:10:11.872|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:11.872|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00076/
|2020-10-02 13:10:11.894|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:11.904|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:11.955|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-944733| - DM workflow kickoff done
|2020-10-02 13:10:11.955|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-944733| - b'id=f4728db6-103b-42eb-9881-d7a185e01a9c owner=8idiuser status=pending startTime=1601662211.94 startTimestamp=2020/10/02 13:10:11 CDT \n'
|2020-10-02 13:10:11.968|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:12.058|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:12.059|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:12.197|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00076 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:12.834|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:12.954|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:16.509|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-944825| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00076/C027_CPMV_att00_00076_0001-100000.hdf
|2020-10-02 13:10:16.765|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00077
|2020-10-02 13:10:16.765|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:16.766|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00077/
|2020-10-02 13:10:16.787|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:16.799|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:16.841|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-944825| - DM workflow kickoff done
|2020-10-02 13:10:16.841|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-944825| - b'id=30c1a84f-f295-484c-a5a5-c2f3b53710d7 owner=8idiuser status=pending startTime=1601662216.82 startTimestamp=2020/10/02 13:10:16 CDT \n'
|2020-10-02 13:10:16.871|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:16.959|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:16.960|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:17.073|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00077 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:17.624|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:17.740|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:21.721|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-944917| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00077/C027_CPMV_att00_00077_0001-100000.hdf
|2020-10-02 13:10:21.971|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00078
|2020-10-02 13:10:21.971|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:21.971|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00078/
|2020-10-02 13:10:21.991|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:22.002|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:22.057|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-944917| - DM workflow kickoff done
|2020-10-02 13:10:22.057|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-944917| - b'id=ac9d2548-b8ff-48b6-a464-f432b9f174d9 owner=8idiuser status=pending startTime=1601662222.04 startTimestamp=2020/10/02 13:10:22 CDT \n'
|2020-10-02 13:10:22.066|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:22.121|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:22.121|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:22.230|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00078 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:22.717|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:22.830|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:26.603|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-945009| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00078/C027_CPMV_att00_00078_0001-100000.hdf
|2020-10-02 13:10:26.867|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00079
|2020-10-02 13:10:26.867|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:26.868|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00079/
|2020-10-02 13:10:26.891|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:26.902|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:26.968|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-945009| - DM workflow kickoff done
|2020-10-02 13:10:26.969|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-945009| - b'id=b9e97680-c836-4611-9ce1-d0c5d5a4b5b0 owner=8idiuser status=pending startTime=1601662226.95 startTimestamp=2020/10/02 13:10:26 CDT \n'
|2020-10-02 13:10:26.972|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:27.059|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:27.059|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:27.211|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00079 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:27.695|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:27.820|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:31.414|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-945101| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00079/C027_CPMV_att00_00079_0001-100000.hdf
|2020-10-02 13:10:31.671|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00080
|2020-10-02 13:10:31.672|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:31.672|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00080/
|2020-10-02 13:10:31.697|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:31.707|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:31.754|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-945101| - DM workflow kickoff done
|2020-10-02 13:10:31.754|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-945101| - b'id=fadd0266-f1eb-4943-b0bc-aa9ae1093f5c owner=8idiuser status=pending startTime=1601662231.74 startTimestamp=2020/10/02 13:10:31 CDT \n'
|2020-10-02 13:10:31.772|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:31.831|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:31.831|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:31.935|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00080 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:32.462|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:32.584|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:36.284|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-945193| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00080/C027_CPMV_att00_00080_0001-100000.hdf
|2020-10-02 13:10:36.522|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00081
|2020-10-02 13:10:36.522|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:36.522|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00081/
|2020-10-02 13:10:36.565|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:36.575|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:36.614|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:36.614|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-945193| - DM workflow kickoff done
|2020-10-02 13:10:36.614|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-945193| - b'id=83f98891-f81a-4379-a025-2eecaaa2d7a1 owner=8idiuser status=pending startTime=1601662236.6 startTimestamp=2020/10/02 13:10:36 CDT \n'
|2020-10-02 13:10:36.677|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:36.678|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:36.789|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00081 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:37.178|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:37.295|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:41.068|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-945285| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00081/C027_CPMV_att00_00081_0001-100000.hdf
|2020-10-02 13:10:41.327|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00082
|2020-10-02 13:10:41.327|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:41.327|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00082/
|2020-10-02 13:10:41.376|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:41.389|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:41.401|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-945285| - DM workflow kickoff done
|2020-10-02 13:10:41.401|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-945285| - b'id=ac392483-1351-46c8-9917-002b8ab68464 owner=8idiuser status=pending startTime=1601662241.38 startTimestamp=2020/10/02 13:10:41 CDT \n'
|2020-10-02 13:10:41.462|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:41.523|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:41.523|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:41.647|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00082 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:42.231|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:42.337|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:46.007|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-945377| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00082/C027_CPMV_att00_00082_0001-100000.hdf
|2020-10-02 13:10:46.273|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00083
|2020-10-02 13:10:46.273|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:46.273|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00083/
|2020-10-02 13:10:46.296|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:46.309|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:46.383|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:46.404|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-945377| - DM workflow kickoff done
|2020-10-02 13:10:46.405|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-945377| - b'id=578fe38c-bd5d-4b57-aa0d-10491361c631 owner=8idiuser status=pending startTime=1601662246.36 startTimestamp=2020/10/02 13:10:46 CDT \n'
|2020-10-02 13:10:46.473|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:46.474|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:46.583|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00083 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:46.975|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:47.097|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:50.683|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-945469| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00083/C027_CPMV_att00_00083_0001-100000.hdf
|2020-10-02 13:10:50.923|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00084
|2020-10-02 13:10:50.923|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:50.923|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00084/
|2020-10-02 13:10:50.968|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:50.977|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:51.017|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:51.022|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-945469| - DM workflow kickoff done
|2020-10-02 13:10:51.022|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-945469| - b'id=b3db76b1-bf80-47ad-9383-08924e39a37a owner=8idiuser status=pending startTime=1601662251.0 startTimestamp=2020/10/02 13:10:50 CDT \n'
|2020-10-02 13:10:51.081|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:51.081|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:51.190|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo FILE:F:C027_CPMV_att00_00084 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=43, #connections=11, #context switches=2]
|2020-10-02 13:10:51.677|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:51.799|INFO|20292|bluesky-session|rigaku_ufxc|68|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=4, #open fd=41, #threads=34, #connections=11, #context switches=2]
|2020-10-02 13:10:55.524|INFO|20292|bluesky-session|xpcs_acquire|308|Thread-945561| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00084/C027_CPMV_att00_00084_0001-100000.hdf
|2020-10-02 13:10:55.850|INFO|20292|bluesky-session|xpcs_acquire|313|Thread-945561| - DM workflow kickoff done
|2020-10-02 13:10:55.850|INFO|20292|bluesky-session|xpcs_acquire|314|Thread-945561| - b'id=f6ce473b-7e5f-407f-b05c-73138c03a3fc owner=8idiuser status=pending startTime=1601662255.83 startTimestamp=2020/10/02 13:10:55 CDT \n'
|2020-10-02 13:10:56.776|INFO|20292|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C027_CPMV_att00_00085
|2020-10-02 13:10:56.776|INFO|20292|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 13:10:56.777|INFO|20292|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C027_CPMV_att00_00085/
|2020-10-02 13:10:56.813|INFO|20292|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 13:10:56.831|INFO|20292|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 13:10:56.932|INFO|20292|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 13:10:57.033|INFO|20292|bluesky-session|rigaku_ufxc|122|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 13:10:57.034|INFO|20292|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 13:10:57.034|ERROR|20292|bluesky-session|rigaku_ufxc|39|bluesky-run-engine| - process_info status: sleeping

Kacperswitalski · 2020-10-02T20:36:33Z

@prjemian C031 crashed at measurement 00047 at 3:26 pm on 10/02

Error message from terminal

HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00037/C031_CPMV_att00_00037_0001-100000.hdf  
C031_CPMV_att00_00038
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00038/C031_CPMV_att00_00038_0001-100000.hdf  
C031_CPMV_att00_00039
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00039/C031_CPMV_att00_00039_0001-100000.hdf  
C031_CPMV_att00_00040
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00040/C031_CPMV_att00_00040_0001-100000.hdf  
C031_CPMV_att00_00041
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00041/C031_CPMV_att00_00041_0001-100000.hdf  
C031_CPMV_att00_00042
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00042/C031_CPMV_att00_00042_0001-100000.hdf  
C031_CPMV_att00_00043                                                                                                    
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00043/C031_CPMV_att00_00043_0001-100000.hdf  
C031_CPMV_att00_00044
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00044/C031_CPMV_att00_00044_0001-100000.hdf  
C031_CPMV_att00_00045
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00045/C031_CPMV_att00_00045_0001-100000.hdf  
C031_CPMV_att00_00046
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00046/C031_CPMV_att00_00046_0001-100000.hdf  
C031_CPMV_att00_00047
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 0.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV',att_value=0,acq_rep=40000,samx_cen=2.2,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    238         yield from bps.open_run(md=md)
    239         for obj in devices:
--> 240             yield from bps.stage(obj)
    241         grp = bps._short_uid('trigger')
    242         no_wait = True

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in stage(obj)
    694     :func:`bluesky.plan_stubs.unstage`
    695     """
--> 696     return (yield Msg('stage', obj))
    697 
    698 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _stage(self, msg)
   2166         if not hasattr(obj, 'stage'):
   2167             return []
-> 2168         result = obj.stage()
   2169         self._staged.add(obj)  # add first in case of failure below
   2170         await self._reset_checkpoint_state_coro()

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in stage(self)
    227         shutter_control.put("Open")
    228         cmd = f"echo FILE:F:{self.batch_name.get()} | nc rigaku1.xray.aps.anl.gov 10000"
--> 229         self.unix_process.put(cmd)
    230 
    231     def trigger(self):

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in put(self, unix_command)
     96 
     97     def put(self, unix_command):
---> 98         status = self.set(unix_command)
     99         while not status.done:
    100             time.sleep(0.001)

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in set(self, unix_command)
     75             stderr = subprocess.PIPE,
     76             )
---> 77         pi = get_process_info(self.process.pid)
     78         logger.info(
     79             "%s [#open files=%d, #open fd=%d, #threads=%d, #connections=%d, #context switches=%d]",

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info(pid)
     43     return dict(
     44         memory_info=pd["memory_info"],
---> 45         num_connections=len(pd["connections"]),
     46         num_context_switches=len(pd['num_ctx_switches']),
     47         num_file_descriptors=pd['num_fds'],

TypeError: object of type 'NoneType' has no len()

Logger

|2020-10-02 15:25:32.412|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:32.418|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C031_CPMV_att00_00040 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:32.690|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:32.700|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:36.129|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-3804| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00040/C031_CPMV_att00_00040_0001-100000.hdf
|2020-10-02 15:25:36.320|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C031_CPMV_att00_00041
|2020-10-02 15:25:36.320|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 15:25:36.320|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00041/
|2020-10-02 15:25:36.361|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-3804| - DM workflow kickoff done
|2020-10-02 15:25:36.361|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-3804| - b'id=ae7b308b-70aa-41ad-a6fc-323824052332 owner=8idiuser status=pending startTime=1601670336.34 startTimestamp=2020/10/02 15:25:36 CDT \n'
|2020-10-02 15:25:36.370|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 15:25:36.383|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 15:25:36.428|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 15:25:36.490|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 15:25:36.506|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:36.508|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C031_CPMV_att00_00041 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:36.856|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:36.860|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:40.377|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-3896| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00041/C031_CPMV_att00_00041_0001-100000.hdf
|2020-10-02 15:25:40.568|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C031_CPMV_att00_00042
|2020-10-02 15:25:40.568|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 15:25:40.568|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00042/
|2020-10-02 15:25:40.587|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 15:25:40.597|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 15:25:40.599|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-3896| - DM workflow kickoff done
|2020-10-02 15:25:40.599|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-3896| - b'id=ee63ed5c-90df-4a9d-8f3f-6c32e685f28e owner=8idiuser status=pending startTime=1601670340.58 startTimestamp=2020/10/02 15:25:40 CDT \n'
|2020-10-02 15:25:40.659|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 15:25:40.712|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 15:25:40.725|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:40.728|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C031_CPMV_att00_00042 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=4, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:41.070|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:41.074|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:44.473|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-3988| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00042/C031_CPMV_att00_00042_0001-100000.hdf
|2020-10-02 15:25:44.677|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-3988| - DM workflow kickoff done
|2020-10-02 15:25:44.677|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-3988| - b'id=44573042-f0ee-4e11-a943-e4db8499047c owner=8idiuser status=pending startTime=1601670344.66 startTimestamp=2020/10/02 15:25:44 CDT \n'
|2020-10-02 15:25:45.673|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C031_CPMV_att00_00043
|2020-10-02 15:25:45.674|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 15:25:45.674|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00043/
|2020-10-02 15:25:45.707|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 15:25:45.720|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 15:25:45.790|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 15:25:45.886|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 15:25:45.901|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:45.904|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C031_CPMV_att00_00043 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=5, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:46.255|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:46.262|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:49.556|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-4080| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00043/C031_CPMV_att00_00043_0001-100000.hdf
|2020-10-02 15:25:49.719|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C031_CPMV_att00_00044
|2020-10-02 15:25:49.719|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 15:25:49.719|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00044/
|2020-10-02 15:25:49.744|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-4080| - DM workflow kickoff done
|2020-10-02 15:25:49.744|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-4080| - b'id=9ee43cb4-f363-4544-a43e-455eb1642d0d owner=8idiuser status=pending startTime=1601670349.73 startTimestamp=2020/10/02 15:25:49 CDT \n'
|2020-10-02 15:25:49.765|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 15:25:49.776|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 15:25:49.815|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 15:25:49.871|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 15:25:49.884|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:49.886|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C031_CPMV_att00_00044 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=5, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:50.240|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:50.248|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:53.661|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-4172| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00044/C031_CPMV_att00_00044_0001-100000.hdf
|2020-10-02 15:25:53.863|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C031_CPMV_att00_00045
|2020-10-02 15:25:53.863|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 15:25:53.864|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00045/
|2020-10-02 15:25:53.884|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 15:25:53.892|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-4172| - DM workflow kickoff done
|2020-10-02 15:25:53.892|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-4172| - b'id=a3a9c8b2-32c5-4545-8cf5-da4472b44726 owner=8idiuser status=pending startTime=1601670353.87 startTimestamp=2020/10/02 15:25:53 CDT \n'
|2020-10-02 15:25:53.896|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 15:25:53.959|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 15:25:54.019|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 15:25:54.033|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:54.036|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C031_CPMV_att00_00045 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=4, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:54.348|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:54.357|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:58.112|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-4264| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00045/C031_CPMV_att00_00045_0001-100000.hdf
|2020-10-02 15:25:58.304|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C031_CPMV_att00_00046
|2020-10-02 15:25:58.304|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 15:25:58.304|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00046/
|2020-10-02 15:25:58.308|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-4264| - DM workflow kickoff done
|2020-10-02 15:25:58.308|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-4264| - b'id=a6a212b8-5bc3-4f3d-8414-29ec0bf96fbb owner=8idiuser status=pending startTime=1601670358.29 startTimestamp=2020/10/02 15:25:58 CDT \n'
|2020-10-02 15:25:58.325|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 15:25:58.364|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 15:25:58.402|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 15:25:58.476|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 15:25:58.504|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: sleeping
|2020-10-02 15:25:58.514|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C031_CPMV_att00_00046 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:25:58.871|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:25:58.877|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 15:26:02.258|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-4356| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00046/C031_CPMV_att00_00046_0001-100000.hdf
|2020-10-02 15:26:02.419|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C031_CPMV_att00_00047
|2020-10-02 15:26:02.419|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 15:26:02.419|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C031_CPMV_att00_00047/
|2020-10-02 15:26:02.463|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 15:26:02.473|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 15:26:02.511|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 15:26:02.572|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 15:26:02.587|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 15:26:02.913|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-4356| - DM workflow kickoff done
|2020-10-02 15:26:02.913|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-4356| - b'id=03a67b97-66ba-448c-9b31-329be944fb50 owner=8idiuser status=pending startTime=1601670362.89 startTimestamp=2020/10/02 15:26:02 CDT \n'

prjemian · 2020-10-02T21:41:53Z

Ok, failed at a different point. Since this is diagnostic code, perhaps we can avoid this.

Kacperswitalski · 2020-10-02T21:43:33Z

@prjemian C032 crashed at measurement 00709 at 4:27 pm on 10/02

Error message from terminal

C032_CPMV_att00_00700
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00700/C032_CPMV_att00_00700_0001-100000.hdf
C032_CPMV_att00_00701
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00701/C032_CPMV_att00_00701_0001-100000.hdf
C032_CPMV_att00_00702
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00702/C032_CPMV_att00_00702_0001-100000.hdf
C032_CPMV_att00_00703
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00703/C032_CPMV_att00_00703_0001-100000.hdf
C032_CPMV_att00_00704
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00704/C032_CPMV_att00_00704_0001-100000.hdf
C032_CPMV_att00_00705
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00705/C032_CPMV_att00_00705_0001-100000.hdf
C032_CPMV_att00_00706
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00706/C032_CPMV_att00_00706_0001-100000.hdf
C032_CPMV_att00_00707
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00707/C032_CPMV_att00_00707_0001-100000.hdf
C032_CPMV_att00_00708
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00708/C032_CPMV_att00_00708_0001-100000.hdf
C032_CPMV_att00_00709
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 0.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV',att_value=0,acq_rep=40000,samx_cen=2.2,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    238         yield from bps.open_run(md=md)
    239         for obj in devices:
--> 240             yield from bps.stage(obj)
    241         grp = bps._short_uid('trigger')
    242         no_wait = True

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in stage(obj)
    694     :func:`bluesky.plan_stubs.unstage`
    695     """
--> 696     return (yield Msg('stage', obj))
    697 
    698 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _stage(self, msg)
   2166         if not hasattr(obj, 'stage'):
   2167             return []
-> 2168         result = obj.stage()
   2169         self._staged.add(obj)  # add first in case of failure below
   2170         await self._reset_checkpoint_state_coro()

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in stage(self)
    227         shutter_control.put("Open")
    228         cmd = f"echo FILE:F:{self.batch_name.get()} | nc rigaku1.xray.aps.anl.gov 10000"
--> 229         self.unix_process.put(cmd)
    230 
    231     def trigger(self):

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in put(self, unix_command)
     96 
     97     def put(self, unix_command):
---> 98         status = self.set(unix_command)
     99         while not status.done:
    100             time.sleep(0.001)

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in set(self, unix_command)
     75             stderr = subprocess.PIPE,
     76             )
---> 77         pi = get_process_info(self.process.pid)
     78         logger.info(
     79             "%s [#open files=%d, #open fd=%d, #threads=%d, #connections=%d, #context switches=%d]",

~/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py in get_process_info(pid)
     43     return dict(
     44         memory_info=pd["memory_info"],
---> 45         num_connections=len(pd["connections"]),
     46         num_context_switches=len(pd['num_ctx_switches']),
     47         num_file_descriptors=pd['num_fds'],

TypeError: object of type 'NoneType' has no len()

Logger

|2020-10-02 16:27:06.566|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 16:27:06.622|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 16:27:06.649|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:06.651|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C032_CPMV_att00_00702 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:06.934|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:06.937|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:10.322|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-69048| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00702/C032_CPMV_att00_00702_0001-100000.hdf
|2020-10-02 16:27:10.520|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C032_CPMV_att00_00703
|2020-10-02 16:27:10.521|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 16:27:10.521|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00703/
|2020-10-02 16:27:10.528|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-69048| - DM workflow kickoff done
|2020-10-02 16:27:10.528|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-69048| - b'id=81f2430d-7cb0-4130-b0e6-47a11f5b9090 owner=8idiuser status=pending startTime=1601674030.51 startTimestamp=2020/10/02 16:27:10 CDT \n'
|2020-10-02 16:27:10.565|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 16:27:10.574|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 16:27:10.611|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 16:27:10.712|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 16:27:10.753|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:10.757|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C032_CPMV_att00_00703 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:11.033|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:11.036|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:14.360|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-69140| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00703/C032_CPMV_att00_00703_0001-100000.hdf
|2020-10-02 16:27:14.554|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-69140| - DM workflow kickoff done
|2020-10-02 16:27:14.555|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-69140| - b'id=05703d00-2655-4f43-b1fe-d5b8da2cca46 owner=8idiuser status=pending startTime=1601674034.54 startTimestamp=2020/10/02 16:27:14 CDT \n'
|2020-10-02 16:27:14.562|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C032_CPMV_att00_00704
|2020-10-02 16:27:14.562|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 16:27:14.562|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00704/
|2020-10-02 16:27:14.581|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 16:27:14.591|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 16:27:14.659|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 16:27:14.724|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 16:27:14.756|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:14.759|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C032_CPMV_att00_00704 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=4, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:15.130|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:15.133|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:18.570|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-69232| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00704/C032_CPMV_att00_00704_0001-100000.hdf
|2020-10-02 16:27:18.769|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-69232| - DM workflow kickoff done
|2020-10-02 16:27:18.769|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-69232| - b'id=3eaf2354-867d-4c51-acdb-82b666c88965 owner=8idiuser status=pending startTime=1601674038.75 startTimestamp=2020/10/02 16:27:18 CDT \n'
|2020-10-02 16:27:18.771|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C032_CPMV_att00_00705
|2020-10-02 16:27:18.771|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 16:27:18.772|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00705/
|2020-10-02 16:27:18.792|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 16:27:18.806|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 16:27:18.877|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 16:27:18.958|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 16:27:18.987|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:18.989|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C032_CPMV_att00_00705 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:19.349|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:19.352|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:22.664|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-69324| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00705/C032_CPMV_att00_00705_0001-100000.hdf
|2020-10-02 16:27:22.866|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C032_CPMV_att00_00706
|2020-10-02 16:27:22.866|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 16:27:22.867|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00706/
|2020-10-02 16:27:22.903|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-69324| - DM workflow kickoff done
|2020-10-02 16:27:22.903|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-69324| - b'id=eefffe91-9fa1-4944-8eda-30fbf02a66b0 owner=8idiuser status=pending startTime=1601674042.87 startTimestamp=2020/10/02 16:27:22 CDT \n'
|2020-10-02 16:27:22.907|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 16:27:22.922|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 16:27:22.988|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 16:27:23.058|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 16:27:23.089|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:23.094|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C032_CPMV_att00_00706 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:23.464|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:23.469|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:27.058|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-69416| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00706/C032_CPMV_att00_00706_0001-100000.hdf
|2020-10-02 16:27:27.267|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C032_CPMV_att00_00707
|2020-10-02 16:27:27.267|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 16:27:27.267|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00707/
|2020-10-02 16:27:27.276|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-69416| - DM workflow kickoff done
|2020-10-02 16:27:27.276|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-69416| - b'id=ecc2ecb2-a531-4788-8d6d-e6cf994c0d7a owner=8idiuser status=pending startTime=1601674047.26 startTimestamp=2020/10/02 16:27:27 CDT \n'
|2020-10-02 16:27:27.292|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 16:27:27.305|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 16:27:27.375|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 16:27:27.458|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 16:27:27.488|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:27.490|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C032_CPMV_att00_00707 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:27.873|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:27.878|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:31.373|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-69508| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00707/C032_CPMV_att00_00707_0001-100000.hdf
|2020-10-02 16:27:31.566|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-69508| - DM workflow kickoff done
|2020-10-02 16:27:31.566|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-69508| - b'id=f77a49b5-c6b7-4e5a-ad79-3a12f4b3c2e1 owner=8idiuser status=pending startTime=1601674051.55 startTimestamp=2020/10/02 16:27:31 CDT \n'
|2020-10-02 16:27:31.571|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C032_CPMV_att00_00708
|2020-10-02 16:27:31.571|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 16:27:31.572|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00708/
|2020-10-02 16:27:31.592|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 16:27:31.602|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 16:27:31.664|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 16:27:31.761|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 16:27:31.799|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:31.803|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo FILE:F:C032_CPMV_att00_00708 | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:32.182|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:32.185|INFO|20024|bluesky-session|rigaku_ufxc|78|bluesky-run-engine| - echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 [#open files=0, #open fd=3, #threads=1, #connections=0, #context switches=2]
|2020-10-02 16:27:35.586|INFO|20024|bluesky-session|xpcs_acquire|308|Thread-69600| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00708/C032_CPMV_att00_00708_0001-100000.hdf
|2020-10-02 16:27:35.790|INFO|20024|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - C032_CPMV_att00_00709
|2020-10-02 16:27:35.790|INFO|20024|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-02 16:27:35.790|INFO|20024|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/C032_CPMV_att00_00709/
|2020-10-02 16:27:35.812|INFO|20024|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-02 16:27:35.825|INFO|20024|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-02 16:27:35.897|INFO|20024|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-02 16:27:35.963|INFO|20024|bluesky-session|rigaku_ufxc|124|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-02 16:27:35.995|INFO|20024|bluesky-session|rigaku_ufxc|35|bluesky-run-engine| - process_info status: running
|2020-10-02 16:27:36.207|INFO|20024|bluesky-session|xpcs_acquire|313|Thread-69600| - DM workflow kickoff done
|2020-10-02 16:27:36.207|INFO|20024|bluesky-session|xpcs_acquire|314|Thread-69600| - b'id=4cbc46bc-af30-4829-b9e9-811a92c79d6b owner=8idiuser status=pending startTime=1601674056.19 startTimestamp=2020/10/02 16:27:36 CDT \n'

Kacperswitalski · 2020-10-02T21:53:35Z

py
I Fri-16:52:53 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/tables.py
I Fri-16:52:53 - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/lambda_750k.py
[TerminalIPythonApp] WARNING | Unknown error in handling startup files:
File "/home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py", line 50
)
^
SyntaxError: unmatched ')'

Kacperswitalski · 2020-10-03T13:55:31Z

@prjemian C034 crashed at measurement 08539 at 4:21 am on 10/03

Error from terminal


C034_CPMV_att00_08524
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08524/C034_CPMV_att00_08524_0001-100000.hdf
C034_CPMV_att00_08525
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08525/C034_CPMV_att00_08525_0001-100000.hdf
C034_CPMV_att00_08526
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08526/C034_CPMV_att00_08526_0001-100000.hdf
C034_CPMV_att00_08527                                                                               
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08527/C034_CPMV_att00_08527_0001-100000.hdf
C034_CPMV_att00_08528
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08528/C034_CPMV_att00_08528_0001-100000.hdf
C034_CPMV_att00_08529
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08529/C034_CPMV_att00_08529_0001-100000.hdf
C034_CPMV_att00_08530
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08530/C034_CPMV_att00_08530_0001-100000.hdf
C034_CPMV_att00_08531
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08531/C034_CPMV_att00_08531_0001-100000.hdf
C034_CPMV_att00_08532
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08532/C034_CPMV_att00_08532_0001-100000.hdf
C034_CPMV_att00_08533
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08533/C034_CPMV_att00_08533_0001-100000.hdf
C034_CPMV_att00_08534
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08534/C034_CPMV_att00_08534_0001-100000.hdf
C034_CPMV_att00_08535
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08535/C034_CPMV_att00_08535_0001-100000.hdf
C034_CPMV_att00_08536
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08536/C034_CPMV_att00_08536_0001-100000.hdf
C034_CPMV_att00_08537
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08537/C034_CPMV_att00_08537_0001-100000.hdf
C034_CPMV_att00_08538
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08538/C034_CPMV_att00_08538_0001-100000.hdf
C034_CPMV_att00_08539
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('S:FillNumber') timed out after 2.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
---------------------------------------------------------------------------
ReadTimeoutError                          Traceback (most recent call last)
<ipython-input-2-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 0.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV',att_value=0,acq_rep=40000,samx_cen=2.2,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    236 
    237     def inner_count(devices, md={}):
--> 238         yield from bps.open_run(md=md)
    239         for obj in devices:
    240             yield from bps.stage(obj)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in open_run(md)
    821     :func:`bluesky.plans_stubs.close_run`
    822     """
--> 823     return (yield Msg('open_run', **(md or {})))
    824 
    825 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in trigger_and_read(devices, name)
    909         return ret
    910     from .preprocessors import rewindable_wrapper
--> 911     return (yield from rewindable_wrapper(inner_trigger_and_read(),
    912                                           rewindable))
    913 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in rewindable_wrapper(plan, rewindable)
    691                                             restore_rewindable()))
    692     else:
--> 693         return (yield from plan)
    694 
    695 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in inner_trigger_and_read()
    903         ret = {}  # collect and return readings to give plan access to them
    904         for obj in devices:
--> 905             reading = (yield from read(obj))
    906             if reading is not None:
    907                 ret.update(reading)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in read(obj)
     93         Msg('read', obj)
     94     """
---> 95     return (yield Msg('read', obj))
     96 
     97 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _read(self, msg)
   1668         obj = msg.obj
   1669         # actually _read_ the object
-> 1670         ret = obj.read(*msg.args, **msg.kwargs)
   1671 
   1672         if ret is None:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/device.py in read(self)
   1235 
   1236         for _, component in self._get_components_of_kind(Kind.normal):
-> 1237             res.update(component.read())
   1238         return res
   1239 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/utils/epics_pvs.py in wrapper(self, *args, **kwargs)
    196     def wrapper(self, *args, **kwargs):
    197         if self.connected:
--> 198             return fcn(self, *args, **kwargs)
    199         else:
    200             raise DisconnectedError('{} is not connected'.format(self.name))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in read(self)
    364             dict
    365         '''
--> 366         value = self.get()
    367         return {self.name: {'value': value,
    368                             'timestamp': self.timestamp}}

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in get(self, as_string, timeout, connection_timeout, form, **kwargs)
   1093             as_string = self._string
   1094 
-> 1095         info = self._get_with_timeout(
   1096             self._read_pv, timeout, connection_timeout, as_string, form)
   1097 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in _get_with_timeout(self, pv, timeout, connection_timeout, as_string, form)
   1056 
   1057         if info is None:
-> 1058             raise ReadTimeoutError(
   1059                 f"Failed to read {pv.pvname} "
   1060                 f"within {timeout:.2} sec")

ReadTimeoutError: Failed to read S:FillNumber within 2.0 sec

prjemian · 2020-10-03T14:36:35Z

Ok, that's a different failure (read timeout after 2.0 seconds). Ironic that the read failed while recording metadata (the storage ring fill number) that is not essential to the measurement or analysis, only useful for diagnostics later. ☹️

On the brighter side, that we ran this far with no error from psutil means this particular issue was fixed with commit 4d764f7. I'll close this issue after creating a new issue for the new problem.

qzhang234 assigned prjemian Sep 29, 2020

qzhang234 self-assigned this Sep 29, 2020

qzhang234 added bug Something isn't working high priority labels Sep 29, 2020

qzhang234 changed the title ~~Bluesky Crashed after 2000 measurements~~ Bluesky crash due to psutil.NoSuchProcess when sending filename to Rigaku Sep 29, 2020

qzhang234 changed the title ~~Bluesky crash due to psutil.NoSuchProcess when sending filename to Rigaku~~ Bluesky crash: psutil.NoSuchProcess when sending filename to Rigaku Sep 29, 2020

qzhang234 assigned sureshnaps and Kacperswitalski and unassigned sureshnaps Oct 2, 2020

prjemian pushed a commit that referenced this issue Oct 2, 2020

ref #231 get it working today with quieter diagnostics

4d764f7

prjemian mentioned this issue Oct 3, 2020

Read timeout stopped measurements. #233

Closed

prjemian closed this as completed Oct 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bluesky crash: psutil.NoSuchProcess when sending filename to Rigaku #231

Bluesky crash: psutil.NoSuchProcess when sending filename to Rigaku #231

qzhang234 commented Sep 29, 2020

prjemian commented Sep 29, 2020

prjemian commented Sep 29, 2020 •

edited

Loading

qzhang234 commented Sep 29, 2020

prjemian commented Sep 29, 2020

prjemian commented Sep 29, 2020

qzhang234 commented Sep 29, 2020

qzhang234 commented Sep 29, 2020

qzhang234 commented Sep 29, 2020

prjemian commented Sep 29, 2020

qzhang234 commented Sep 29, 2020 •

edited

Loading

qzhang234 commented Sep 29, 2020 •

edited

Loading

qzhang234 commented Sep 29, 2020

qzhang234 commented Sep 29, 2020

prjemian commented Oct 1, 2020

qzhang234 commented Oct 2, 2020 •

edited

Loading

Kacperswitalski commented Oct 2, 2020 •

edited by qzhang234

Loading

Kacperswitalski commented Oct 2, 2020

prjemian commented Oct 2, 2020

Kacperswitalski commented Oct 2, 2020

Kacperswitalski commented Oct 2, 2020

Kacperswitalski commented Oct 3, 2020

prjemian commented Oct 3, 2020

Bluesky crash: psutil.NoSuchProcess when sending filename to Rigaku #231

Bluesky crash: psutil.NoSuchProcess when sending filename to Rigaku #231

Comments

qzhang234 commented Sep 29, 2020

prjemian commented Sep 29, 2020

prjemian commented Sep 29, 2020 • edited Loading

qzhang234 commented Sep 29, 2020

prjemian commented Sep 29, 2020

prjemian commented Sep 29, 2020

qzhang234 commented Sep 29, 2020

qzhang234 commented Sep 29, 2020

qzhang234 commented Sep 29, 2020

prjemian commented Sep 29, 2020

qzhang234 commented Sep 29, 2020 • edited Loading

qzhang234 commented Sep 29, 2020 • edited Loading

qzhang234 commented Sep 29, 2020

qzhang234 commented Sep 29, 2020

prjemian commented Oct 1, 2020

qzhang234 commented Oct 2, 2020 • edited Loading

Kacperswitalski commented Oct 2, 2020 • edited by qzhang234 Loading

Kacperswitalski commented Oct 2, 2020

prjemian commented Oct 2, 2020

Kacperswitalski commented Oct 2, 2020

Kacperswitalski commented Oct 2, 2020

Kacperswitalski commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Sep 29, 2020 •

edited

Loading

qzhang234 commented Sep 29, 2020 •

edited

Loading

qzhang234 commented Sep 29, 2020 •

edited

Loading

qzhang234 commented Oct 2, 2020 •

edited

Loading

Kacperswitalski commented Oct 2, 2020 •

edited by qzhang234

Loading