Lambda Time Out: IOC Says 'Too Many Open Files' #146

qzhang234 · 2020-02-07T03:52:43Z

As in #132...

Error message on Bluesky:

In [65]: RE(lambda_test(num_iter=200, acquire_time=0.004, acquire_period=0.004, num_images=1200, sample_name="WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_1p5mm", sample_
    ...: prefix="A", sample_suffix="Lq0", analysis_true_false=True))                                                                                            
I Thu-21:26:01 - Moving samx to 1.75, samz to -1.15
samplestage_z:  30%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d                                                                      | 0.04525/0.15 [00:00<00:01, 13.28s/mm]
samplestage_x:  94%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588      | 0.23525/0.25 [00:00<00:00,  2.56s/mm]
                                                                                                                                                                
                                                                                                                                                                
I Thu-21:26:03 - AD_Acquire starting
I Thu-21:26:03 - file_path = /home/8-id-i/2020-1/richards202002/A067_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_1p5mm_Lq0_001/
I Thu-21:26:03 - Shutter will *REMAIN OPEN THROUGH* the Acquisition...
I Thu-21:26:03 - calling full_acquire_procedure()
D Thu-21:26:03 - before update_metadata_prescan()
I Thu-21:26:03 - detNum=25, det_pars={'ccdHardwareRowSize': 516, 'ccdHardwareColSize': 1556, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.055, 'saturation': 4095, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 1, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'LAMBDA'}
D Thu-21:26:03 - after update_metadata_prescan()
---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
<ipython-input-65-12445712bd15> in <module>
----> 1 RE(lambda_test(num_iter=200, acquire_time=0.004, acquire_period=0.004, num_images=1200, sample_name="WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_1p5mm", sample_prefix="A", sample_suffix="Lq0", analysis_true_false=True))

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    742                     # it (unless it is a canceled error)
    743                     if exc is not None:
--> 744                         raise exc
    745 
    746             if self._interrupted:

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1251             self.log.error("Run aborted")
   1252             self.log.error("%r", err)
-> 1253             raise err
   1254         finally:
   1255             # Some done_callbacks may still be alive in other threads.

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1138                     else:
   1139                         try:
-> 1140                             msg = self._plan_stack[-1].send(resp)
   1141                         # We have exhausted the top generator
   1142                         except StopIteration:

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1292         plan = monitor_during_wrapper(plan, self.monitors)
   1293         plan = baseline_wrapper(plan, self.baseline)
-> 1294         return (yield from plan)

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1147         return (yield from plan)
   1148     else:
-> 1149         return (yield from plan_mutator(plan, insert_baseline))
   1150 
   1151 

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    788     plan1 = plan_mutator(plan, insert_after_open)
    789     plan2 = plan_mutator(plan1, insert_before_close)
--> 790     return (yield from plan2)
    791 
    792 

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    846     plan1 = plan_mutator(plan, insert_after_open)
    847     plan2 = plan_mutator(plan1, insert_before_close)
--> 848     return (yield from plan2)
    849 
    850 

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    167                     continue
    168                 else:
--> 169                     raise ex
    170         # if inserting / mutating, put new generator on the stack
    171         # and replace the current msg with the first element from the

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    120             ret = result_stack.pop()
    121             try:
--> 122                 msg = plan_stack[-1].send(ret)
    123             except StopIteration as e:
    124                 # discard the exhausted generator

~/.ipython-bluesky/user/lambda_testing.py in lambda_test(num_iter, acquire_time, acquire_period, num_images, sample_name, sample_prefix, sample_suffix, analysis_true_false)
     55             submit_xpcs_job=analysis_true_false,
     56             atten=None, path='/home/8-id-i/2020-1/richards202002/',
---> 57             md={"sample_name": sample_name})
     58         logger.info("-"*20 + " end of acquire")
     59 

~/.ipython-bluesky/profile_bluesky/startup/50-controls.py in AD_Acquire(areadet, acquire_time, acquire_period, num_images, file_name, submit_xpcs_job, atten, path, md)
   1071 
   1072     logger.info("calling full_acquire_procedure()")
-> 1073     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1011                 plan = gen_func(*inner_args, **inner_kwargs)
   1012                 plan = wrapper(plan, *args, **kwargs)
-> 1013                 return (yield from plan)
   1014             return dec_inner
   1015         return dec

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    937         return (yield from plan)
    938 
--> 939     return (yield from finalize_wrapper(inner(), unstage_devices()))
    940 
    941 

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    498     cleanup = True
    499     try:
--> 500         ret = yield from plan
    501     except GeneratorExit:
    502         cleanup = False

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in inner()
    935     def inner():
    936         yield from stage_devices()
--> 937         return (yield from plan)
    938 
    939     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1011                 plan = gen_func(*inner_args, **inner_kwargs)
   1012                 plan = wrapper(plan, *args, **kwargs)
-> 1013                 return (yield from plan)
   1014             return dec_inner
   1015         return dec

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    788     plan1 = plan_mutator(plan, insert_after_open)
    789     plan2 = plan_mutator(plan1, insert_before_close)
--> 790     return (yield from plan2)
    791 
    792 

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     75             # if we have a stashed exception, pass it along
     76             try:
---> 77                 msg = plan_stack[-1].throw(exception)
     78             except StopIteration as e:
     79                 # discard the exhausted generator

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     75             # if we have a stashed exception, pass it along
     76             try:
---> 77                 msg = plan_stack[-1].throw(exception)
     78             except StopIteration as e:
     79                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/50-controls.py in full_acquire_procedure(md)
   1039         logger.debug("before count()")
   1040         # yield from bp.count([areadet], md=md)
-> 1041         yield from inner_count([areadet], md=_md)
   1042         logger.debug("after count()")
   1043 

~/.ipython-bluesky/profile_bluesky/startup/50-controls.py in inner_count(devices, md)
    996         yield from bps.open_run(md=md)
    997         for obj in devices:
--> 998             yield from bps.stage(obj)
    999         grp = bps._short_uid('trigger')
   1000         no_wait = True

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/plan_stubs.py in stage(obj)
    584     :func:`bluesky.plan_stubs.unstage`
    585     """
--> 586     return (yield Msg('stage', obj))
    587 
    588 

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    192         try:
    193             # yield out the 'current message' and collect the return
--> 194             inner_ret = yield msg
    195         except GeneratorExit:
    196             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1195                         # exceptions (coming in via throw) can be
   1196                         # raised
-> 1197                         new_response = yield from coro(msg)
   1198 
   1199                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky/lib/python3.7/site-packages/bluesky/run_engine.py in _stage(self, msg)
   2202         if not hasattr(obj, 'stage'):
   2203             return []
-> 2204         result = obj.stage()
   2205         self._staged.add(obj)  # add first in case of failure below
   2206         yield from self._reset_checkpoint_state_coro()

~/.ipython-bluesky/profile_bluesky/startup/26-lambda.py in stage(self)
    516 
    517     def stage(self):
--> 518         super().stage()
    519         root = os.path.join("/", "home", "8-id-i/")
    520         if self._file_path.startswith("/data/"):

/home/beams10/8IDIUSER/Documents/jemian/github/ophyd/ophyd/device.py in stage(self)
    557                 device = getattr(self, attr)
    558                 if hasattr(device, 'stage'):
--> 559                     device.stage()
    560                     devices_staged.append(device)
    561         except Exception:

/home/beams10/8IDIUSER/Documents/jemian/github/ophyd/ophyd/device.py in stage(self)
    548                                self.name,
    549                                val, original_vals[sig])
--> 550                 set_and_wait(sig, val)
    551                 # It worked -- now add it to this list of sigs to unstage.
    552                 self._original_vals[sig] = original_vals[sig]

/home/beams10/8IDIUSER/Documents/jemian/github/ophyd/ophyd/utils/epics_pvs.py in set_and_wait(signal, val, poll_time, timeout, rtol, atol)
    265             raise TimeoutError("Attempted to set %r to value %r and timed "
    266                                "out after %r seconds. Current value is %r." %
--> 267                                (signal, val, timeout, current_value))
    268 
    269 

TimeoutError: Attempted to set EpicsSignalWithRBV(read_pv='8LAMBDA1:IMMout:Capture_RBV', name='lambdadet_immout_capture', parent='lambdadet_immout', value=0, timestamp=1581043409.489933, auto_monitor=False, string=False, write_pv='8LAMBDA1:IMMout:Capture', limits=False, put_complete=False) to value 1 and timed out after 10 seconds. Current value is 0.

The text was updated successfully, but these errors were encountered:

qzhang234 · 2020-02-07T03:53:08Z

Error message on Lambda IOC:

IMM Recurse path: statis = 0
2020/02/06 20:43:29.216 Lambda:writeInt32 Setting TriggerMode 0
2020/02/06 20:43:29.217 Lambda:writeInt32 Setting TriggerMode 0
2020/02/06 20:43:29.218 Entering Lambda:writeFloat64
2020/02/06 20:43:29.219 Entering Lambda:writeFloat64
Setting Acquire Period
mkdir umask 777 chmod ret 0
 Created Dir /data/2020-1/richards202002/A066_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_Lq0_178
IMM Recurse path: statis = 0
Opened IMM File
Error- Could not open IMM File
2020/02/06 20:43:29.490 NDPluginFile::openFileBase Error opening file /data/2020-1/richards202002/A066_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_Lq0_178/A066_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_Lq0_178.imm, status=3
IMM closeFile
IMM Recurse path: statis = 0
2020/02/06 20:43:30.265 Lambda:writeInt32 Setting TriggerMode 0
2020/02/06 20:43:30.267 Lambda:writeInt32 Setting TriggerMode 0
2020/02/06 20:43:30.268 Entering Lambda:writeFloat64
2020/02/06 20:43:30.269 Entering Lambda:writeFloat64
Setting Acquire Period
mkdir umask 777 chmod ret 0
 Created Dir /data/2020-1/richards202002/A066_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_Lq0_179
IMM Recurse path: statis = 0
Error- Could not open IMM File
2020/02/06 20:43:30.488 NDPluginFile::openFileBase Error opening file /data/2020-1/richards202002/A066_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_Lq0_179/A066_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_Lq0_179.imm, status=3
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-204337]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-204337]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-204337]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
IMM Recurse path: statis = 0
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-204407]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-204407]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-204407]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-204507]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-204507]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-204507]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-204607]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-204607]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-204607]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-204707]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-204707]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-204707]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-204807]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-204807]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-204807]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-204907]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-204907]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-204907]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205007]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205007]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205007]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205107]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205107]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205107]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205207]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205207]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205207]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205307]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205307]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-205307]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205407]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205407]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205407]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205507]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205507]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205507]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205607]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205607]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205607]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205707]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205707]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205707]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205807]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205807]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205807]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-205907]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-205907]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-205907]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210007]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210007]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-210007]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210107]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210107]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-210107]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210207]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210207]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-210207]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210307]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210307]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-210307]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210407]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210407]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-210407]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210507]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210507]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-210507]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210607]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210607]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-210607]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210707]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210707]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-210707]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210807]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210807]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-210807]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-210907]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-210907]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-210907]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211007]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211007]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-211007]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211107]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211107]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-211107]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211207]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211207]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-211207]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211307]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211307]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-211307]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211407]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211407]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-211407]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211507]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211507]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-211507]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211607]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211607]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-211607]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211707]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211707]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-211707]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211807]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211807]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-211807]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-211907]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-211907]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-211907]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212007]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212007]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-212007]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212107]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212107]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-212107]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212207]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212207]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-212207]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212307]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212307]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-212307]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212407]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212407]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-212407]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212507]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212507]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-212507]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2020/02/06 21:26:03.626 Lambda:writeInt32 Setting TriggerMode 0
2020/02/06 21:26:03.629 Lambda:writeInt32 Setting TriggerMode 0
2020/02/06 21:26:03.631 Entering Lambda:writeFloat64
2020/02/06 21:26:03.634 Entering Lambda:writeFloat64
Setting Acquire Period
mkdir umask 777 chmod ret 0
 Created Dir /data/2020-1/richards202002/A067_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_1p5mm_Lq0_001
IMM Recurse path: statis = 0
Error- Could not open IMM File
2020/02/06 21:26:03.883 NDPluginFile::openFileBase Error opening file /data/2020-1/richards202002/A067_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_1p5mm_Lq0_001/A067_WLM_75mgpml_RbCl_1p0_W3p2_att1_15C_1p5mm_Lq0_001.imm, status=3
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212604]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212604]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-212604]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
IMM Recurse path: statis = 0
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212634]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212634]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-212634]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212734]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212734]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-212734]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212834]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212834]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-212834]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-212934]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-212934]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-212934]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213034]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213034]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-213034]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213134]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213134]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-213134]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213234]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213234]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-213234]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213334]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213334]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-213334]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213434]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213434]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-213434]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213534]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213534]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-213534]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213634]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213634]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-213634]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213734]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213734]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-213734]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213834]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213834]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-213834]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-213934]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-213934]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-213934]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214034]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214034]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214034]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214134]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214134]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214134]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214234]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214234]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214234]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214334]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214334]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214334]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214434]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214434]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214434]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214534]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214534]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Can't write new backup file. [200206-214534]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214634]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214634]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214634]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214734]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214734]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214734]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214834]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214834]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214834]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-214934]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-214934]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-214934]
../save_restore.c(1704): [0x18]=write_it:Too many open files
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200206-215034]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200206-215034]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200206-215034]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1704): [0x18]=write_it:Too many open files

qzhang234 · 2020-02-07T03:53:55Z

Lambda still shows that it is acquiring at the time when user reported Bluesky failure:

qzhang234 · 2020-02-07T04:06:58Z

I pressed 'Stop' on the EPICS panel, but same error shows up again when I rerun the plan. As a result I rebooted Bluesky and Lambda and everything seems to be running, however it would be critical to resolve this problem

prjemian · 2020-02-07T04:46:50Z

Thanks for writing this up. I'll work with the group to identify and resolve the issue. Might take some patience.

…

On Thu, Feb 6, 2020, 10:10 PM qzhang234 ***@***.***> wrote: Assigned #146 <#146> to @prjemian <https://github.com/prjemian>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#146?email_source=notifications&email_token=AARMUMF2LGIEJQPA62XSYKLRBTNMNA5CNFSM4KRIAJBKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWPEVI7Q#event-3016316030>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMUMBTY2B2A3WCTU3MT2DRBTNMNANCNFSM4KRIAJBA> .

prjemian · 2020-02-07T18:16:40Z

Is it the OS version (or OS updates)?

Suspecting that updates to the operating system might be the cause, we booted with the OS kernel that was used in mid-December. Stlil, we observe many open files.

Answer: No, not the OS version

prjemian · 2020-02-07T18:19:19Z

What about SPEC? Does it see so many open files?

Restart the IOC before making this test, run similar acquisition from SPEC. For a short time, we see a couple dozen open .imm files at any time, but shortly after acquisition is complete, that number drops to zero.

Answer: No problem with SPEC.

Conclusion: our attention is focused on what is different between how we use this detector from SPEC and bluesky.

prjemian · 2020-02-07T18:21:11Z

Using bluesky, we see that 21 files .imm remain open on the EPICS IOC console (lsof | grep LambdaApp | grep imm | sort -u | wc -l) for each time we call AD_Acquire(). This increase in open files with each acquisition ultimately results in the error reported above.

prjemian · 2020-02-07T18:26:39Z

Answer: No, not the OS version

Later today. we'll ask IT to reboot with the most recent kernel.

prjemian · 2020-02-07T18:39:01Z

Comparing the GUI screens when SPEC uses the detector and bluesky, we see no differences (aside from the file name and path).

prjemian · 2020-02-07T18:43:16Z

@keenanlang : What can we do as an EPICS client that would compel the IMMout plugin to keep a file open on the IOC?

@mrakitin, @danielballan, @tacaswell: What can we do from bluesky that would compel the IMMout plugin to keep a file open on the IOC?

prjemian · 2020-02-10T19:03:22Z

Install various updates to bluesky environment including

versions    =========== =================   
            key         value               
            =========== =================   
            apstools    1.1.18+2.g260a7c4   
            bluesky     1.5.5               
            databroker  0.13.3              
            epics       3.4.1               
            matplotlib  3.1.3               
            numpy       1.18.1              
            ophyd       1.4.0rc4            
            pyRestTable 2020.0.2            
            spec2nexus  2021.1.3            
            =========== =================

epics-base 7.0.3
matplotlib update
ipython update

prjemian · 2020-02-10T19:04:08Z

With updates in the framework, see if we still have a problem from bluesky acquisition.

prjemian · 2020-02-10T19:08:46Z

Run 1 acquisition loop from bluesky:

In [1]: %run -m lambda_testing                                                                                                                                             
I Mon-13:05:13 - /home/beams10/8IDIUSER/.ipython-bluesky/user/lambda_testing.py

In [2]: RE(lambda_test(num_iter=1, acquire_time=0.001, acquire_period=0.001, num_images=1000, sample_name="bluesky_test_with_sleep", sample_prefix="PJ", sample_suffix="Lq0
   ...: ", analysis_true_false=False))                                                                                                                                     
I Mon-13:06:30 - Moving samx to 0.0, samz to 0.0
samplestage_z:  68%|███████████████████████████████████████████████████████████████████████████▋                                    | 0.38525/0.57 [00:03<00:01, 10.36s/mm]
samplestage_x:  99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏| 1.8865/1.9 [00:03<00:00,  2.07s/mm]
                                                                                                                                                                           
                                                                                                                                                                           
I Mon-13:06:36 - AD_Acquire starting
I Mon-13:06:36 - file_path = /home/8-id-i/2020-1/bluesky/PJ079_bluesky_test_with_sleep_Lq0_001/
I Mon-13:06:36 - Shutter will *REMAIN OPEN THROUGH* the Acquisition...
I Mon-13:06:36 - calling full_acquire_procedure()
D Mon-13:06:36 - before update_metadata_prescan()
I Mon-13:06:36 - detNum=25, det_pars={'ccdHardwareRowSize': 516, 'ccdHardwareColSize': 1556, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.055, 'saturation': 4095, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 1, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'LAMBDA'}
D Mon-13:06:36 - after update_metadata_prescan()
I Mon-13:06:36 - scaler should be autocounting now
D Mon-13:06:36 - before count()
I Mon-13:06:36 - full_name: /home/8-id-i/2020-1/bluesky/PJ079_bluesky_test_with_sleep_Lq0_001/PJ079_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
D Mon-13:06:37 - lambdadet.cam.state=0
D Mon-13:06:37 - old value=None
D Mon-13:06:37 - lambdadet.cam.state=0
D Mon-13:06:37 - capture=1
D Mon-13:06:37 - old value=0
D Mon-13:06:37 - capture=1
D Mon-13:06:37 - lambdadet.cam.state=4
lambdadet [In progress. No progress bar available.]                                                                                                                        
D Mon-13:06:38 - lambdadet.cam.state=5
D Mon-13:06:38 - old value=4
D Mon-13:06:38 - capture=1
D Mon-13:06:38 - closed shutter
I Mon-13:06:38 - watch_acquire() method ends
I Mon-13:06:38 - cam.acquire.get()=0
I Mon-13:06:38 - immout.capture.get()=0
lambdadet [Complete.]                                                                                                                                                      
                                                                                                                                                                           
HDF5 workflow file name: /home/8-id-i/2020-1/bluesky/PJ079_bluesky_test_with_sleep_Lq0_001/PJ079_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-13:06:39 - DM workflow kickoff starting: analysis:False  file:/home/8-id-i/2020-1/bluesky/PJ079_bluesky_test_with_sleep_Lq0_001/PJ079_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-13:06:39 - diagnostic sleep
I Mon-13:06:41 - -------------------- end of acquire
Out[2]: ('f77286b1-73bb-49a4-b5e6-0f1040d2bbf0',)

In [3]: I Mon-13:06:42 - DM workflow kickoff done                                                                                                                          
I Mon-13:06:42 - b'id=ee7af66c-1e8e-4426-a349-07742c37ae9e owner=8idiuser status=pending startTime=1581361602.24 startTimestamp=2020/02/10 13:06:42 CST \n'

Analysis from lsof command above is that 21 files are open.

Now, exit bluesky, and check lsof again. Once bluesky is exited, the lsof command still shows 21 files open.

prjemian · 2020-02-10T19:48:18Z

Being more systematic about diagnosing this, have written new function to print out all the EpicsSignal values in a single ophyd device. Here is an example:

In [8]: object_explorer(lambdadet)                                                                                                                                                                    
41
========================= ============================================ ===================================================================================================
name                      PV reference                                 value                                                                                              
========================= ============================================ ===================================================================================================
cam.acquire               8LAMBDA1:cam1:Acquire_RBV                    0                                                                                                  
cam.acquire_period        8LAMBDA1:cam1:AcquirePeriod_RBV              0.001                                                                                              
cam.acquire_time          8LAMBDA1:cam1:AcquireTime_RBV                0.001                                                                                              
cam.array_callbacks       8LAMBDA1:cam1:ArrayCallbacks_RBV             1                                                                                                  
cam.array_size_x          8LAMBDA1:cam1:ArraySizeX_RBV                 1556                                                                                               
cam.array_size_y          8LAMBDA1:cam1:ArraySizeY_RBV                 516                                                                                                
cam.bad_frame_counter     8LAMBDA1:cam1:BadFrameCounter                0                                                                                                  
cam.config_file_path      8LAMBDA1:cam1:ConfigFilePath                 /local/epics/synApps_5_8/support/areaDetector-R3-2/ADLambda/iocs/LambdaIOC/iocBoot/iocLambda/config
cam.data_type             8LAMBDA1:cam1:DataType_RBV                   3                                                                                                  
cam.detector_state        8LAMBDA1:cam1:DetectorState_RBV              Idle                                                                                               
cam.firmware_version      8LAMBDA1:cam1:FirmwareVersion_RBV            unknown                                                                                            
cam.image_mode            8LAMBDA1:cam1:ImageMode_RBV                  1                                                                                                  
cam.num_images            8LAMBDA1:cam1:NumImages_RBV                  1000                                                                                               
cam.operating_mode        8LAMBDA1:cam1:OperatingMode_RBV              0                                                                                                  
cam.serial_number         8LAMBDA1:cam1:SerialNumber_RBV               505J10                                                                                             
cam.state                 8LAMBDA1:cam1:LambdaState                    ON                                                                                                 
cam.status_msg            8LAMBDA1:cam1:StatusMessage_RBV                                                                                                                 
cam.temperature           8LAMBDA1:cam1:Temperature_RBV                25.0                                                                                               
cam.trigger_mode          8LAMBDA1:cam1:TriggerMode                    0                                                                                                  
imm0.capture              8LAMBDA1:IMM0:Capture_RBV                    1                                                                                                  
imm0.file_format          8LAMBDA1:IMM0:NDFileIMM_format_RBV           IMM_Cmprs                                                                                          
imm0.num_captured         8LAMBDA1:IMM0:NumCaptured_RBV                334                                                                                                
imm1.capture              8LAMBDA1:IMM1:Capture_RBV                    1                                                                                                  
imm1.file_format          8LAMBDA1:IMM1:NDFileIMM_format_RBV           IMM_Cmprs                                                                                          
imm1.num_captured         8LAMBDA1:IMM1:NumCaptured_RBV                333                                                                                                
imm2.capture              8LAMBDA1:IMM2:Capture_RBV                    1                                                                                                  
imm2.file_format          8LAMBDA1:IMM2:NDFileIMM_format_RBV           IMM_Cmprs                                                                                          
imm2.num_captured         8LAMBDA1:IMM2:NumCaptured_RBV                333                                                                                                
immout.blocking_callbacks 8LAMBDA1:IMMout:BlockingCallbacks_RBV        1                                                                                                  
immout.capture            8LAMBDA1:IMMout:Capture_RBV                  0                                                                                                  
immout.enable             8LAMBDA1:IMMout:EnableCallbacks_RBV          Enable                                                                                             
immout.file_format        8LAMBDA1:IMMout:NDFileIMM_format_RBV         IMM_Cmprs                                                                                          
immout.file_name          8LAMBDA1:IMMout:FileName_RBV                 A119_Test_000C_att0_Lq0_002                                                                        
immout.file_number        8LAMBDA1:IMMout:FileNumber_RBV               1001                                                                                               
immout.file_path          8LAMBDA1:IMMout:FilePath_RBV                 /data/2020-1/sanat202002/A119_Test_000C_att0_Lq0_002/                                              
immout.full_file_name     8LAMBDA1:IMMout:FullFileName_RBV             /data/2020-1/sanat202002/A119_Test_000C_att0_Lq0_002/A119_Test_000C_att0_Lq0_002_00001-01000.imm   
immout.num_capture        8LAMBDA1:IMMout:NumCapture_RBV               1000                                                                                               
immout.num_captured       8LAMBDA1:IMMout:NumCaptured_RBV              1000                                                                                               
immout.num_pixels         8LAMBDA1:IMMout:NDFileIMM_num_imm_pixels_RBV 9                                                                                                  
immout.unique_id          8LAMBDA1:IMMout:NDFileIMM_uniqueID_RBV       9962                                                                                               
stats1.mean_value         8LAMBDA1:Stats1:MeanValue_RBV                0.0006974751399932245                                                                              
========================= ============================================ ===================================================================================================

prjemian · 2020-02-10T19:48:54Z

To be clear, the files we finding open after bluesky acquisitions are the image files from the IMMout plugin, the .imm files.

prjemian · 2020-02-10T20:18:58Z

LambdaApp 26458       8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26468 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26469 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26470 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26471 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26472 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26473 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26474 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26475 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26476 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26477 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26478 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26479 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26480 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26481 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26482 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26483 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26484 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26485 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26486 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
LambdaApp 26458 26487 8idiuser   31w      REG    8,16   1098622 8660927595 /data/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm

1 parent file process
20 child file processes

These are the files acquired from the bluesky plan.

prjemian · 2020-02-10T20:19:10Z

Interesting...

reboot Lambda IOC
pump image through the plugin chain
start bluesky
run one acquisition from bluesky
note number of open files on IOC is 21
run one acquisition from MEDM using the settings left in the EPICS PVs
note number of open files on IOC is still 21 (no new open files)

prjemian · 2020-02-10T20:25:38Z

Bluesky session console from today:

details

                                                                                                                                                                           
I Mon-13:06:36 - AD_Acquire starting
I Mon-13:06:36 - file_path = /home/8-id-i/2020-1/bluesky/PJ079_bluesky_test_with_sleep_Lq0_001/
I Mon-13:06:36 - Shutter will *REMAIN OPEN THROUGH* the Acquisition...
I Mon-13:06:36 - calling full_acquire_procedure()
D Mon-13:06:36 - before update_metadata_prescan()
I Mon-13:06:36 - detNum=25, det_pars={'ccdHardwareRowSize': 516, 'ccdHardwareColSize': 1556, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.055, 'saturation': 4095, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 1, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'LAMBDA'}
D Mon-13:06:36 - after update_metadata_prescan()
I Mon-13:06:36 - scaler should be autocounting now
D Mon-13:06:36 - before count()
I Mon-13:06:36 - full_name: /home/8-id-i/2020-1/bluesky/PJ079_bluesky_test_with_sleep_Lq0_001/PJ079_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
D Mon-13:06:37 - lambdadet.cam.state=0
D Mon-13:06:37 - old value=None
D Mon-13:06:37 - lambdadet.cam.state=0
D Mon-13:06:37 - capture=1
D Mon-13:06:37 - old value=0
D Mon-13:06:37 - capture=1
D Mon-13:06:37 - lambdadet.cam.state=4
lambdadet [In progress. No progress bar available.]                                                                                                                        
D Mon-13:06:38 - lambdadet.cam.state=5
D Mon-13:06:38 - old value=4
D Mon-13:06:38 - capture=1
D Mon-13:06:38 - closed shutter
I Mon-13:06:38 - watch_acquire() method ends
I Mon-13:06:38 - cam.acquire.get()=0
I Mon-13:06:38 - immout.capture.get()=0
lambdadet [Complete.]                                                                                                                                                      
                                                                                                                                                                           
HDF5 workflow file name: /home/8-id-i/2020-1/bluesky/PJ079_bluesky_test_with_sleep_Lq0_001/PJ079_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-13:06:39 - DM workflow kickoff starting: analysis:False  file:/home/8-id-i/2020-1/bluesky/PJ079_bluesky_test_with_sleep_Lq0_001/PJ079_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-13:06:39 - diagnostic sleep
I Mon-13:06:41 - -------------------- end of acquire
Out[2]: ('f77286b1-73bb-49a4-b5e6-0f1040d2bbf0',)

In [3]: I Mon-13:06:42 - DM workflow kickoff done                                                                                                                          
I Mon-13:06:42 - b'id=ee7af66c-1e8e-4426-a349-07742c37ae9e owner=8idiuser status=pending startTime=1581361602.24 startTimestamp=2020/02/10 13:06:42 CST \n'
In [3]: exit                                                                                                                                                               

(bluesky) 8idiuser@quartz ~/.ipython-bluesky $ bluesky8IDI 
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

IPython profile: bluesky
/home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/00-0-checks.py
...
In [3]: %run -m lambda_testing                                                                                                                                                                        
I Mon-13:29:24 - /home/beams10/8IDIUSER/.ipython-bluesky/user/lambda_testing.py

In [4]: RE(lambda_test(num_iter=1, acquire_time=0.001, acquire_period=0.001, num_images=1000, sample_name="bluesky_test_with_sleep", sample_prefix="PJ", sample_suffix="Lq0", analysis_true_false=Fals
   ...: e))                                                                                                                                                                                           
I Mon-13:29:32 - Moving samx to 0.0, samz to 0.0
I Mon-13:29:32 - AD_Acquire starting
I Mon-13:29:32 - file_path = /home/8-id-i/2020-1/bluesky/PJ080_bluesky_test_with_sleep_Lq0_001/
I Mon-13:29:32 - Shutter will *REMAIN OPEN THROUGH* the Acquisition...
I Mon-13:29:32 - calling full_acquire_procedure()
D Mon-13:29:32 - before update_metadata_prescan()
I Mon-13:29:32 - detNum=25, det_pars={'ccdHardwareRowSize': 516, 'ccdHardwareColSize': 1556, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.055, 'saturation': 4095, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 1, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'LAMBDA'}
D Mon-13:29:32 - after update_metadata_prescan()
I Mon-13:29:32 - scaler should be autocounting now
D Mon-13:29:32 - before count()
I Mon-13:29:33 - full_name: /home/8-id-i/2020-1/bluesky/PJ080_bluesky_test_with_sleep_Lq0_001/PJ080_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
D Mon-13:29:34 - lambdadet.cam.state=0
D Mon-13:29:34 - lambdadet.cam.state=0
D Mon-13:29:34 - old value=ON
D Mon-13:29:34 - old value=0
D Mon-13:29:34 - capture=1
D Mon-13:29:34 - capture=1
D Mon-13:29:34 - lambdadet.cam.state=4
D Mon-13:29:34 - old value=0
D Mon-13:29:34 - capture=1
lambdadet [In progress. No progress bar available.]                                                                                                                                                   
D Mon-13:29:35 - lambdadet.cam.state=5
D Mon-13:29:35 - old value=4
D Mon-13:29:35 - capture=1
D Mon-13:29:36 - closed shutter
I Mon-13:29:36 - watch_acquire() method ends
I Mon-13:29:36 - cam.acquire.get()=0
I Mon-13:29:36 - immout.capture.get()=0
lambdadet [Complete.]                                                                                                                                                                                 
                                                                                                                                                                                                      
HDF5 workflow file name: /home/8-id-i/2020-1/bluesky/PJ080_bluesky_test_with_sleep_Lq0_001/PJ080_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-13:29:36 - DM workflow kickoff starting: analysis:False  file:/home/8-id-i/2020-1/bluesky/PJ080_bluesky_test_with_sleep_Lq0_001/PJ080_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-13:29:36 - diagnostic sleep
I Mon-13:29:37 - DM workflow kickoff done
I Mon-13:29:37 - b'id=4d36ac33-6ed5-48c5-bd0c-e9e0daceab9d owner=8idiuser status=pending startTime=1581362977.01 startTimestamp=2020/02/10 13:29:37 CST \n'
I Mon-13:29:38 - -------------------- end of acquire
Out[4]: ('6ad63509-c508-4192-b6ae-75fb5b3b5ccd',)

In [5]: object_explorer(lambdadet)                                                                                                                                                                    
41
========================= ============================================ ================================================================================================================
name                      PV reference                                 value                                                                                                           
========================= ============================================ ================================================================================================================
cam.acquire               8LAMBDA1:cam1:Acquire_RBV                    0                                                                                                               
cam.acquire_period        8LAMBDA1:cam1:AcquirePeriod_RBV              0.001                                                                                                           
cam.acquire_time          8LAMBDA1:cam1:AcquireTime_RBV                0.001                                                                                                           
cam.array_callbacks       8LAMBDA1:cam1:ArrayCallbacks_RBV             1                                                                                                               
cam.array_size_x          8LAMBDA1:cam1:ArraySizeX_RBV                 1556                                                                                                            
cam.array_size_y          8LAMBDA1:cam1:ArraySizeY_RBV                 516                                                                                                             
cam.bad_frame_counter     8LAMBDA1:cam1:BadFrameCounter                0                                                                                                               
cam.config_file_path      8LAMBDA1:cam1:ConfigFilePath                 /local/epics/synApps_5_8/support/areaDetector-R3-2/ADLambda/iocs/LambdaIOC/iocBoot/iocLambda/config             
cam.data_type             8LAMBDA1:cam1:DataType_RBV                   3                                                                                                               
cam.detector_state        8LAMBDA1:cam1:DetectorState_RBV              Idle                                                                                                            
cam.firmware_version      8LAMBDA1:cam1:FirmwareVersion_RBV            unknown                                                                                                         
cam.image_mode            8LAMBDA1:cam1:ImageMode_RBV                  1                                                                                                               
cam.num_images            8LAMBDA1:cam1:NumImages_RBV                  1000                                                                                                            
cam.operating_mode        8LAMBDA1:cam1:OperatingMode_RBV              0                                                                                                               
cam.serial_number         8LAMBDA1:cam1:SerialNumber_RBV               505J10                                                                                                          
cam.state                 8LAMBDA1:cam1:LambdaState                    ON                                                                                                              
cam.status_msg            8LAMBDA1:cam1:StatusMessage_RBV                                                                                                                              
cam.temperature           8LAMBDA1:cam1:Temperature_RBV                25.0                                                                                                            
cam.trigger_mode          8LAMBDA1:cam1:TriggerMode                    0                                                                                                               
imm0.capture              8LAMBDA1:IMM0:Capture_RBV                    1                                                                                                               
imm0.file_format          8LAMBDA1:IMM0:NDFileIMM_format_RBV           IMM_Cmprs                                                                                                       
imm0.num_captured         8LAMBDA1:IMM0:NumCaptured_RBV                334                                                                                                             
imm1.capture              8LAMBDA1:IMM1:Capture_RBV                    1                                                                                                               
imm1.file_format          8LAMBDA1:IMM1:NDFileIMM_format_RBV           IMM_Cmprs                                                                                                       
imm1.num_captured         8LAMBDA1:IMM1:NumCaptured_RBV                333                                                                                                             
imm2.capture              8LAMBDA1:IMM2:Capture_RBV                    1                                                                                                               
imm2.file_format          8LAMBDA1:IMM2:NDFileIMM_format_RBV           IMM_Cmprs                                                                                                       
imm2.num_captured         8LAMBDA1:IMM2:NumCaptured_RBV                333                                                                                                             
immout.blocking_callbacks 8LAMBDA1:IMMout:BlockingCallbacks_RBV        1                                                                                                               
immout.capture            8LAMBDA1:IMMout:Capture_RBV                  0                                                                                                               
immout.enable             8LAMBDA1:IMMout:EnableCallbacks_RBV          Enable                                                                                                          
immout.file_format        8LAMBDA1:IMMout:NDFileIMM_format_RBV         IMM_Cmprs                                                                                                       
immout.file_name          8LAMBDA1:IMMout:FileName_RBV                 A118_Test_000C_att0_Lq0_002                                                                                     
immout.file_number        8LAMBDA1:IMMout:FileNumber_RBV               1001                                                                                                            
immout.file_path          8LAMBDA1:IMMout:FilePath_RBV                 /data/2020-1/sanat202002/A118_Test_000C_att0_Lq0_002/                                                           
immout.full_file_name     8LAMBDA1:IMMout:FullFileName_RBV             /data/2020-1/bluesky/PJ080_bluesky_test_with_sleep_Lq0_001/PJ080_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
immout.num_capture        8LAMBDA1:IMMout:NumCapture_RBV               1000                                                                                                            
immout.num_captured       8LAMBDA1:IMMout:NumCaptured_RBV              1000                                                                                                            
immout.num_pixels         8LAMBDA1:IMMout:NDFileIMM_num_imm_pixels_RBV 13                                                                                                              
immout.unique_id          8LAMBDA1:IMMout:NDFileIMM_uniqueID_RBV       6962                                                                                                            
stats1.mean_value         8LAMBDA1:Stats1:MeanValue_RBV                0.0007124210358502222                                                                                           
========================= ============================================ ================================================================================================================

Out[5]: <pyRestTable.rest_table.Table at 0x7f32744a7a58>

In [6]: RE(lambda_test(num_iter=1, acquire_time=0.001, acquire_period=0.001, num_images=1000, sample_name="bluesky_test_with_sleep", sample_prefix="PJ", sample_suffix="Lq0", analysis_true_false=Fals
   ...: e))                                                                                                                                                                                           
I Mon-13:30:48 - Moving samx to 0.1, samz to 0.0
                                                                                                                                                                                                      
                                                                                                                                                                                                      
I Mon-13:30:49 - AD_Acquire starting
I Mon-13:30:49 - file_path = /home/8-id-i/2020-1/bluesky/PJ081_bluesky_test_with_sleep_Lq0_001/
I Mon-13:30:49 - Shutter will *REMAIN OPEN THROUGH* the Acquisition...
I Mon-13:30:49 - calling full_acquire_procedure()
D Mon-13:30:49 - before update_metadata_prescan()
I Mon-13:30:49 - detNum=25, det_pars={'ccdHardwareRowSize': 516, 'ccdHardwareColSize': 1556, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.055, 'saturation': 4095, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 1, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'LAMBDA'}
D Mon-13:30:49 - after update_metadata_prescan()
I Mon-13:30:49 - scaler should be autocounting now
D Mon-13:30:49 - before count()
I Mon-13:30:50 - full_name: /home/8-id-i/2020-1/bluesky/PJ081_bluesky_test_with_sleep_Lq0_001/PJ081_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
D Mon-13:30:50 - lambdadet.cam.state=0
D Mon-13:30:50 - old value=5
D Mon-13:30:50 - capture=1
D Mon-13:30:50 - lambdadet.cam.state=4
lambdadet [In progress. No progress bar available.]                                                                                                                                                   
D Mon-13:30:51 - lambdadet.cam.state=5
D Mon-13:30:51 - old value=4
D Mon-13:30:51 - capture=1
D Mon-13:30:51 - closed shutter
I Mon-13:30:52 - watch_acquire() method ends
I Mon-13:30:52 - cam.acquire.get()=0
I Mon-13:30:52 - immout.capture.get()=0
lambdadet [Complete.]                                                                                                                                                                                 
                                                                                                                                                                                                      
HDF5 workflow file name: /home/8-id-i/2020-1/bluesky/PJ081_bluesky_test_with_sleep_Lq0_001/PJ081_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-13:30:52 - DM workflow kickoff starting: analysis:False  file:/home/8-id-i/2020-1/bluesky/PJ081_bluesky_test_with_sleep_Lq0_001/PJ081_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-13:30:52 - diagnostic sleep
I Mon-13:30:52 - DM workflow kickoff done
I Mon-13:30:52 - b'id=634b29eb-0a36-4b0c-908e-2a5a1cf0cba7 owner=8idiuser status=pending startTime=1581363052.9 startTimestamp=2020/02/10 13:30:52 CST \n'
I Mon-13:30:54 - -------------------- end of acquire
Out[6]: ('a118c450-a511-4b88-9d6d-ce1b2fb9c4f7',)


In [2]: exit                                                                                                                                                                                          
(bluesky) 8idiuser@quartz ~/.ipython-bluesky $ bluesky8IDI 
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

IPython profile: bluesky
/home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/00-0-checks.py
BlueSky version: 1.5.5
Ophyd version: 1.4.0rc4
/home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/00-logging.py
Activating auto-logging. Current session state plus future input saved.
Filename       : /home/beams10/8IDIUSER/.ipython-bluesky/.logs/ipython_console.log
Mode           : rotate
Output logging : True
Raw input log  : False
Timestamping   : True
State          : active
I Mon-14:09:12 - ############################################################ startup
I Mon-14:09:12 - logging started
I Mon-14:09:12 - logging level = 10
I Mon-14:09:12 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/utils/explorer.py
/home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/00-startup-jup.py
Loading metadata history from /home/beams/8IDIUSER/.config/bluesky/bluesky_history.db
I Mon-14:09:12 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/00-startup.py
Loading metadata history from /home/beams/8IDIUSER/.config/bluesky/bluesky_history.db
I Mon-14:09:12 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/01-databroker.py
I Mon-14:09:14 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/02-pyepics.py
I Mon-14:09:14 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/09-imports.py
I Mon-14:09:14 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/09-overrides.py
W Mon-14:09:14 - in 09-overrides.py
W Mon-14:09:14 - APS_utils.show_ophyd_symbols() is available
W Mon-14:09:14 - Can remove local definition show_ophyd_symbols()
I Mon-14:09:14 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/10-devices.py
I Mon-14:09:14 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/11-device-overrides.py
I Mon-14:09:14 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/15-spec-config.py
I Mon-14:09:14 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/20-detectors.py
I Mon-14:09:16 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/21-signals.py
W Mon-14:09:16 - !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
W Mon-14:09:16 - Session started when 8_ID-I is not operating.
W Mon-14:09:16 - Using simulator 'shutter'.
W Mon-14:09:16 - !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I Mon-14:09:16 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/26-lambda.py
/home/beams/8IDIUSER/.conda/envs/bluesky/lib/python3.7/site-packages/dask/config.py:168: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  data = yaml.load(f.read()) or {}
I Mon-14:09:17 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/26-rigaku.py
I Mon-14:09:17 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/35-DM-support.py
I Mon-14:09:17 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/50-controls.py
I Mon-14:09:17 - /home/beams10/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/60-metadata.py
I Mon-14:09:17 - RE metadata = 
=========== ================================
key         value                           
=========== ================================
beamline_id APS TRR 8-ID-I                  
login_id    8idiuser@quartz.xray.aps.anl.gov
pid         3875                            
proposal_id None                            
scan_id     6852                            
versions    =========== =================   
            key         value               
            =========== =================   
            apstools    1.1.18+2.g260a7c4   
            bluesky     1.5.5               
            databroker  0.13.3              
            epics       3.4.1               
            matplotlib  3.1.3               
            numpy       1.18.1              
            ophyd       1.4.0rc4            
            pyRestTable 2020.0.2            
            spec2nexus  2021.1.3            
            =========== =================   
=========== ================================

...
In [1]: object_explorer(lambdadet)                                                                                                                                                                    
41
========================= ============================================ ===================================================================================================
name                      PV reference                                 value                                                                                              
========================= ============================================ ===================================================================================================
cam.acquire               8LAMBDA1:cam1:Acquire_RBV                    0                                                                                                  
cam.acquire_period        8LAMBDA1:cam1:AcquirePeriod_RBV              0.001                                                                                              
cam.acquire_time          8LAMBDA1:cam1:AcquireTime_RBV                0.001                                                                                              
cam.array_callbacks       8LAMBDA1:cam1:ArrayCallbacks_RBV             1                                                                                                  
cam.array_size_x          8LAMBDA1:cam1:ArraySizeX_RBV                 1556                                                                                               
cam.array_size_y          8LAMBDA1:cam1:ArraySizeY_RBV                 516                                                                                                
cam.bad_frame_counter     8LAMBDA1:cam1:BadFrameCounter                0                                                                                                  
cam.config_file_path      8LAMBDA1:cam1:ConfigFilePath                 /local/epics/synApps_5_8/support/areaDetector-R3-2/ADLambda/iocs/LambdaIOC/iocBoot/iocLambda/config
cam.data_type             8LAMBDA1:cam1:DataType_RBV                   3                                                                                                  
cam.detector_state        8LAMBDA1:cam1:DetectorState_RBV              Idle                                                                                               
cam.firmware_version      8LAMBDA1:cam1:FirmwareVersion_RBV            unknown                                                                                            
cam.image_mode            8LAMBDA1:cam1:ImageMode_RBV                  1                                                                                                  
cam.num_images            8LAMBDA1:cam1:NumImages_RBV                  4735                                                                                               
cam.operating_mode        8LAMBDA1:cam1:OperatingMode_RBV              0                                                                                                  
cam.serial_number         8LAMBDA1:cam1:SerialNumber_RBV               505J10                                                                                             
cam.state                 8LAMBDA1:cam1:LambdaState                    ON                                                                                                 
cam.status_msg            8LAMBDA1:cam1:StatusMessage_RBV                                                                                                                 
cam.temperature           8LAMBDA1:cam1:Temperature_RBV                25.0                                                                                               
cam.trigger_mode          8LAMBDA1:cam1:TriggerMode                    0                                                                                                  
imm0.capture              8LAMBDA1:IMM0:Capture_RBV                    1                                                                                                  
imm0.file_format          8LAMBDA1:IMM0:NDFileIMM_format_RBV           IMM_Cmprs                                                                                          
imm0.num_captured         8LAMBDA1:IMM0:NumCaptured_RBV                452                                                                                                
imm1.capture              8LAMBDA1:IMM1:Capture_RBV                    1                                                                                                  
imm1.file_format          8LAMBDA1:IMM1:NDFileIMM_format_RBV           IMM_Cmprs                                                                                          
imm1.num_captured         8LAMBDA1:IMM1:NumCaptured_RBV                453                                                                                                
imm2.capture              8LAMBDA1:IMM2:Capture_RBV                    1                                                                                                  
imm2.file_format          8LAMBDA1:IMM2:NDFileIMM_format_RBV           IMM_Cmprs                                                                                          
imm2.num_captured         8LAMBDA1:IMM2:NumCaptured_RBV                453                                                                                                
immout.blocking_callbacks 8LAMBDA1:IMMout:BlockingCallbacks_RBV        1                                                                                                  
immout.capture            8LAMBDA1:IMMout:Capture_RBV                  0                                                                                                  
immout.enable             8LAMBDA1:IMMout:EnableCallbacks_RBV          Enable                                                                                             
immout.file_format        8LAMBDA1:IMMout:NDFileIMM_format_RBV         IMM_Cmprs                                                                                          
immout.file_name          8LAMBDA1:IMMout:FileName_RBV                 A119_Test_000C_att0_Lq0_002                                                                        
immout.file_number        8LAMBDA1:IMMout:FileNumber_RBV               1001                                                                                               
immout.file_path          8LAMBDA1:IMMout:FilePath_RBV                 /data/2020-1/sanat202002/A119_Test_000C_att0_Lq0_002/                                              
immout.full_file_name     8LAMBDA1:IMMout:FullFileName_RBV                                                                                                                
immout.num_capture        8LAMBDA1:IMMout:NumCapture_RBV               1000                                                                                               
immout.num_captured       8LAMBDA1:IMMout:NumCaptured_RBV              0                                                                                                  
immout.num_pixels         8LAMBDA1:IMMout:NDFileIMM_num_imm_pixels_RBV 14                                                                                                 
immout.unique_id          8LAMBDA1:IMMout:NDFileIMM_uniqueID_RBV       29089                                                                                              
stats1.mean_value         8LAMBDA1:Stats1:MeanValue_RBV                0.000719893983778721                                                                               
========================= ============================================ ===================================================================================================

Out[1]: <pyRestTable.rest_table.Table at 0x7fa8a89de780>

In [2]: %run -m lambda_testing                                                                                                                                                                        
I Mon-14:10:09 - /home/beams10/8IDIUSER/.ipython-bluesky/user/lambda_testing.py

In [3]: RE(lambda_test(num_iter=1, acquire_time=0.001, acquire_period=0.001, num_images=1000, sample_name="bluesky_test_with_sleep", sample_prefix="PJ", sample_suffix="Lq0", analysis_true_false=Fals
   ...: e))                                                                                                                                                                                           
I Mon-14:10:38 - Moving samx to 0.0, samz to 0.0
I Mon-14:10:38 - AD_Acquire starting
I Mon-14:10:38 - file_path = /home/8-id-i/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/
I Mon-14:10:38 - Shutter will *REMAIN OPEN THROUGH* the Acquisition...
I Mon-14:10:38 - calling full_acquire_procedure()
D Mon-14:10:38 - before update_metadata_prescan()
I Mon-14:10:38 - detNum=25, det_pars={'ccdHardwareRowSize': 516, 'ccdHardwareColSize': 1556, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.055, 'saturation': 4095, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 1, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'LAMBDA'}
D Mon-14:10:38 - after update_metadata_prescan()
I Mon-14:10:38 - scaler should be autocounting now
D Mon-14:10:38 - before count()
I Mon-14:10:39 - full_name: /home/8-id-i/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_00001-01000.imm
D Mon-14:10:39 - lambdadet.cam.state=0
D Mon-14:10:39 - lambdadet.cam.state=0
D Mon-14:10:39 - old value=ON
D Mon-14:10:39 - old value=0
D Mon-14:10:39 - capture=1
D Mon-14:10:39 - capture=1
D Mon-14:10:39 - lambdadet.cam.state=4
D Mon-14:10:39 - old value=0
D Mon-14:10:39 - capture=1
lambdadet [In progress. No progress bar available.]                                                                                                                                                   
D Mon-14:10:40 - lambdadet.cam.state=5
D Mon-14:10:40 - old value=4
D Mon-14:10:40 - capture=1
D Mon-14:10:41 - closed shutter
I Mon-14:10:42 - watch_acquire() method ends
I Mon-14:10:42 - cam.acquire.get()=0
I Mon-14:10:42 - immout.capture.get()=0
lambdadet [Complete.]                                                                                                                                                                                 
                                                                                                                                                                                                      
HDF5 workflow file name: /home/8-id-i/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-14:10:43 - DM workflow kickoff starting: analysis:False  file:/home/8-id-i/2020-1/bluesky/PJ082_bluesky_test_with_sleep_Lq0_001/PJ082_bluesky_test_with_sleep_Lq0_001_0001-1000.hdf
I Mon-14:10:43 - diagnostic sleep
I Mon-14:10:43 - DM workflow kickoff done
I Mon-14:10:43 - b'id=0fe8be3b-281f-4626-9c57-f3e28c9a5621 owner=8idiuser status=pending startTime=1581365443.82 startTimestamp=2020/02/10 14:10:43 CST \n'
I Mon-14:10:45 - -------------------- end of acquire
Out[3]: ('93b616bd-3c72-4fde-bd48-f96aae31bd6d',)

prjemian · 2020-02-10T20:27:11Z

From the most recent observation, focus on differences in the configuration of the IMMout plugin from SPEC and Bluesky.

prjemian · 2020-02-11T01:22:20Z

Talked with @keenanlang about the new support he is developing. Since its arrival is imminent, let's suspend further work on this issue since that new support will require a (at least partial) rewrite of 26-lambda.py.

In the interim, I'm working on #148 which should make the enormous number of bluesky support components easier to navigate.

qzhang234 · 2021-03-04T18:16:00Z

@prjemian Has this issue been solved?

prjemian · 2021-06-01T21:14:03Z

Make sure that this issue does not happen with #150 then can close here.

qzhang234 · 2021-06-01T21:17:15Z

The IOC has been updated. Closing the issue for now and commence Bluesky/EPICS testing.

qzhang234 assigned qzhang234, prjemian and sureshnaps and unassigned qzhang234 Feb 7, 2020

qzhang234 added bug Something isn't working high priority labels Feb 7, 2020

prjemian mentioned this issue Feb 7, 2020

use bluesky, et al., for at least one user #46

Closed

prjemian added this to the 2020-02-04 beam time milestone Feb 7, 2020

prjemian pushed a commit that referenced this issue Feb 7, 2020

MNT #146 cleanup

cacac58

prjemian modified the milestones: 2020-02-04 beam time, 2020-03 operations Feb 7, 2020

qzhang234 mentioned this issue Feb 10, 2020

XPCS at APS 8-ID-I Phase 1 lightsources/project-management#59

Open

4 tasks

prjemian pushed a commit that referenced this issue Feb 10, 2020

WIP #146 add diagnostics

f39f500

prjemian pushed a commit that referenced this issue Feb 10, 2020

WIP #146 notes

58ad29f

prjemian mentioned this issue Feb 15, 2020

add object_explorer from APS/8-ID-I BCDA-APS/apstools#288

Closed

prjemian mentioned this issue Jun 29, 2020

Overnight Scan Crashed at 7945th Measurement #188

Closed

qzhang234 closed this as completed Jun 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lambda Time Out: IOC Says 'Too Many Open Files' #146

Lambda Time Out: IOC Says 'Too Many Open Files' #146

qzhang234 commented Feb 7, 2020 •

edited by prjemian

Loading

qzhang234 commented Feb 7, 2020

qzhang234 commented Feb 7, 2020

qzhang234 commented Feb 7, 2020

prjemian commented Feb 7, 2020 via email

prjemian commented Feb 7, 2020

prjemian commented Feb 7, 2020

prjemian commented Feb 7, 2020

prjemian commented Feb 7, 2020

prjemian commented Feb 7, 2020

prjemian commented Feb 7, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020 •

edited

Loading

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 11, 2020

qzhang234 commented Mar 4, 2021

prjemian commented Jun 1, 2021

qzhang234 commented Jun 1, 2021

Lambda Time Out: IOC Says 'Too Many Open Files' #146

Lambda Time Out: IOC Says 'Too Many Open Files' #146

Comments

qzhang234 commented Feb 7, 2020 • edited by prjemian Loading

qzhang234 commented Feb 7, 2020

qzhang234 commented Feb 7, 2020

qzhang234 commented Feb 7, 2020

prjemian commented Feb 7, 2020 via email

prjemian commented Feb 7, 2020

Is it the OS version (or OS updates)?

prjemian commented Feb 7, 2020

What about SPEC? Does it see so many open files?

prjemian commented Feb 7, 2020

prjemian commented Feb 7, 2020

prjemian commented Feb 7, 2020

prjemian commented Feb 7, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020 • edited Loading

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 10, 2020

prjemian commented Feb 11, 2020

qzhang234 commented Mar 4, 2021

prjemian commented Jun 1, 2021

qzhang234 commented Jun 1, 2021

qzhang234 commented Feb 7, 2020 •

edited by prjemian

Loading

prjemian commented Feb 10, 2020 •

edited

Loading