USAXS can't tune motors

[prjemian]

Tuning fails with a puzzling error.

OperationFailure: Index with name: uid_1 already exists with different options

Refactor needed. Due to changes in databroker?

In [10]: RE(preUSAXStune())                                                                                                                 
I Fri-16:36:14 - Preparing for USAXS mode ... please wait ...
I Fri-16:36:15 - Prepared for USAXS mode
                                                                                                                                            
I Fri-16:36:15 - tuning axis: m_stage_r
                                                                                                                                            
D Fri-16:36:15 - gain=1000000.0  rate: 1160.0  max: 950000  converged=[False, True]
D Fri-16:36:15 - gain=10000000.0  rate: 293680.0  max: 950000  converged=[False, True, True, True]
                                                                                                                                            
D Fri-16:36:16 - gain=1000000.0  rate: 112580.0  max: 950000  converged=[True, True]
D Fri-16:36:16 - gain=10000000.0  rate: 292620.0  max: 950000  converged=[True, True, True, True]
D Fri-16:36:16 - gain=1000000000.0  rate: 3120.0  max: 950000  converged=[True, True, True, True, True, True]
D Fri-16:36:16 - converged: [True, True, True, True, True, True]
I Fri-16:36:16 - Tuning axis m_stage_r, current position is 9.051722999999999
Run aborted
Traceback (most recent call last):
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 1355, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1307, in __call__
    return (yield from plan)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1162, in baseline_wrapper
    return (yield from plan_mutator(plan, insert_baseline))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper
    return (yield from plan2)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper
    return (yield from plan2)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/plans/scans.py", line 121, in preUSAXStune
    yield from tune(md=md)
  File "/home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/plans/axis_tuning.py", line 73, in tune_mr
    yield from _tune_base_(m_stage.r, md=md)
  File "/home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/plans/axis_tuning.py", line 51, in _tune_base_
    yield from axis.tune(md=md)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/apstools/devices.py", line 1036, in tune
    yield from self.tuner.tune(md=md, **kwargs)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/apstools/plans.py", line 1085, in tune
    return (yield from _scan(md=_md))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 1045, in dec_inner
    return (yield from plan)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 383, in subs_wrapper
    _unsubscribe()))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 509, in finalize_wrapper
    ret = yield from plan
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 380, in _inner_plan
    return (yield from plan)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/apstools/plans.py", line 1033, in _scan
    yield from bps.open_run(md)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 825, in open_run
    return (yield Msg('open_run', **(md or {})))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator
    msg = plan_stack[-1].throw(exception)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 140, in single_gen
    return (yield msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator
    msg = plan_stack[-1].throw(exception)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 140, in single_gen
    return (yield msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator
    msg = plan_stack[-1].throw(exception)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 140, in single_gen
    return (yield msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 1415, in _run
    new_response = await coro(msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 1600, in _open_run
    new_uid = await current_run.open_run(msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/bundlers.py", line 53, in open_run
    await self.emit(DocumentNames.start, doc)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 2262, in emit
    self.emit_sync(name, doc)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 2259, in emit_sync
    self.dispatcher.process(name, doc)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 2282, in process
    exceptions = self.cb_registry.process(name, name.name, doc)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 351, in process
    func(*args, **kwargs)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 432, in __call__
    return mtd(*args, **kwargs)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/v1.py", line 959, in insert
    if self._serializer is None:
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/v1.py", line 189, in _serializer
    self.__serializer = self._catalog._get_serializer()
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/_drivers/mongo_normalized.py", line 351, in _get_serializer
    return Serializer(self._metadatastore_db, self._asset_registry_db)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/suitcase/mongo_normalized/__init__.py", line 48, in __init__
    self._create_indexes()
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/suitcase/mongo_normalized/__init__.py", line 56, in _create_indexes
    self._resource_collection.create_index('uid')
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py", line 1995, in create_index
    self.__create_index(keys, kwargs, session, **cmd_options)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py", line 1894, in __create_index
    session=session)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py", line 250, in _command
    user_fields=user_fields)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/pool.py", line 613, in command
    user_fields=user_fields)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/network.py", line 167, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/helpers.py", line 159, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
Failed to close run <bluesky.bundlers.RunBundler object at 0x7ff7bc3dbfd0>. different options
---------------------------------------------------------------------------
OperationFailure                          Traceback (most recent call last)
<ipython-input-10-f92c9110a1e3> in <module>
----> 1 RE(preUSAXStune())

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    795             self._task_fut.add_done_callback(set_blocking_event)
    796 
--> 797         self._resume_task(init_func=_build_task)
    798 
    799         if self._interrupted:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    919                     if (exc is not None
    920                             and not isinstance(exc, _RunEnginePanic)):
--> 921                         raise exc
    922 
    923     def install_suspender(self, suspender):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1488             exit_reason = str(err)
   1489             self.log.exception("Run aborted")
-> 1490             raise err
   1491         finally:
   1492             if not exit_reason:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1353                     else:
   1354                         try:
-> 1355                             msg = self._plan_stack[-1].send(resp)
   1356                         # We have exhausted the top generator
   1357                         except StopIteration:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.ipython/profile_bluesky/startup/instrument/plans/scans.py in preUSAXStune(md)
    119     for axis, tune in tuners.items():
    120         yield from bps.mv(ti_filter_shutter, "open")
--> 121         yield from tune(md=md)
    122         if axis.tuner.tune_ok:
    123             # If we don't wait, the next tune often fails

~/.ipython/profile_bluesky/startup/instrument/plans/axis_tuning.py in tune_mr(md)
     71     md['plan_name'] = "tune_mr"
     72     # print(f"metadata={md}")  # TOO much data to print
---> 73     yield from _tune_base_(m_stage.r, md=md)
     74 
     75 

~/.ipython/profile_bluesky/startup/instrument/plans/axis_tuning.py in _tune_base_(axis, md)
     49     )
     50     yield from autoscale_amplifiers([upd_controls, I0_controls, I00_controls])
---> 51     yield from axis.tune(md=md)
     52     yield from bps.mv(
     53         ti_filter_shutter, "close",

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/apstools/devices.py in tune(self, md, **kwargs)
   1034                 yield from self.pre_tune_method()
   1035 
-> 1036             yield from self.tuner.tune(md=md, **kwargs)
   1037 
   1038             if self.post_tune_method is not None:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/apstools/plans.py in tune(self, width, num, md)
   1083             results.report(stream_name)
   1084 
-> 1085         return (yield from _scan(md=_md))
   1086 
   1087 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1043                 plan = gen_func(*inner_args, **inner_kwargs)
   1044                 plan = wrapper(plan, *args, **kwargs)
-> 1045                 return (yield from plan)
   1046             return dec_inner
   1047         return dec

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in subs_wrapper(plan, subs)
    381 
    382     return (yield from finalize_wrapper(_inner_plan(),
--> 383                                         _unsubscribe()))
    384 
    385 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in _inner_plan()
    378     def _inner_plan():
    379         yield from _subscribe()
--> 380         return (yield from plan)
    381 
    382     return (yield from finalize_wrapper(_inner_plan(),

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/apstools/plans.py in _scan(md)
   1031         @bpp.subs_decorator(self.peaks)
   1032         def _scan(md=None):
-> 1033             yield from bps.open_run(md)
   1034 
   1035             position_list = np.linspace(start, finish, num)

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/plan_stubs.py in open_run(md)
    823     :func:`bluesky.plans_stubs.close_run`
    824     """
--> 825     return (yield Msg('open_run', **(md or {})))
    826 
    827 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in single_gen(msg)
    138         the input message
    139     '''
--> 140     return (yield msg)
    141 
    142 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in single_gen(msg)
    138         the input message
    139     '''
--> 140     return (yield msg)
    141 
    142 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in single_gen(msg)
    138         the input message
    139     '''
--> 140     return (yield msg)
    141 
    142 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1413                         # exceptions (coming in via throw) can be
   1414                         # raised
-> 1415                         new_response = await coro(msg)
   1416 
   1417                     # special case `CancelledError` and let the outer

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _open_run(self, msg)
   1598             loop=self.loop)
   1599 
-> 1600         new_uid = await current_run.open_run(msg)
   1601         self._run_start_uids.append(new_uid)
   1602         return new_uid

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/bundlers.py in open_run(self, msg)
     51 
     52         doc = dict(uid=self._run_start_uid, time=ttime.time(), **self._md)
---> 53         await self.emit(DocumentNames.start, doc)
     54         doc_logger.debug("[start] document is emitted (run_uid=%r)", self._run_start_uid,
     55         extra={'doc_name': 'start',

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in emit(self, name, doc)
   2260 
   2261     async def emit(self, name, doc):
-> 2262         self.emit_sync(name, doc)
   2263 
   2264 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in emit_sync(self, name, doc)
   2257         "Process blocking callbacks and schedule non-blocking callbacks."
   2258         schema_validators[name].validate(doc)
-> 2259         self.dispatcher.process(name, doc)
   2260 
   2261     async def emit(self, name, doc):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in process(self, name, doc)
   2280         doc : dict
   2281         """
-> 2282         exceptions = self.cb_registry.process(name, name.name, doc)
   2283         for exc, traceback in exceptions:
   2284             warn("A %r was raised during the processing of a %s "

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in process(self, sig, *args, **kwargs)
    349             for cid, func in list(self.callbacks[sig].items()):
    350                 try:
--> 351                     func(*args, **kwargs)
    352                 except ReferenceError:
    353                     self._remove_proxy(func)

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in __call__(self, *args, **kwargs)
    430             mtd = self.func
    431         # invoke the callable and return the result
--> 432         return mtd(*args, **kwargs)
    433 
    434     def __eq__(self, other):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/v1.py in insert(self, name, doc)
    957 
    958     def insert(self, name, doc):
--> 959         if self._serializer is None:
    960             raise RuntimeError("No Serializer was configured for this.")
    961         warnings.warn(

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/v1.py in _serializer(self)
    187             # which is pending deprecation.
    188             if hasattr(self._catalog, '_get_serializer'):
--> 189                 self.__serializer = self._catalog._get_serializer()
    190         return self.__serializer
    191 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/_drivers/mongo_normalized.py in _get_serializer(self)
    349         "This is used internally by v1.Broker. It may be removed in future."
    350         from suitcase.mongo_normalized import Serializer
--> 351         return Serializer(self._metadatastore_db, self._asset_registry_db)
    352 
    353 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/suitcase/mongo_normalized/__init__.py in __init__(self, metadatastore_db, asset_registry_db)
     46         self._asset_registry_db = assets_db
     47 
---> 48         self._create_indexes()
     49 
     50     def _create_indexes(self):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/suitcase/mongo_normalized/__init__.py in _create_indexes(self)
     54         If the index already exists, this has no effect.
     55         """
---> 56         self._resource_collection.create_index('uid')
     57         self._resource_collection.create_index('resource_id')  # legacy
     58         # TODO: Migrate all Resources to have a RunStart UID, and then make a

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py in create_index(self, keys, session, **kwargs)
   1993         if "maxTimeMS" in kwargs:
   1994             cmd_options["maxTimeMS"] = kwargs.pop("maxTimeMS")
-> 1995         self.__create_index(keys, kwargs, session, **cmd_options)
   1996         return name
   1997 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py in __create_index(self, keys, index_options, session, **kwargs)
   1892                 codec_options=_UNICODE_REPLACE_CODEC_OPTIONS,
   1893                 write_concern=self._write_concern_for(session),
-> 1894                 session=session)
   1895 
   1896     def create_index(self, keys, session=None, **kwargs):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py in _command(self, sock_info, command, slave_ok, read_preference, codec_options, check, allowable_errors, read_concern, write_concern, collation, session, retryable_write, user_fields)
    248                 client=self.__database.client,
    249                 retryable_write=retryable_write,
--> 250                 user_fields=user_fields)
    251 
    252     def __create(self, options, collation, session):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/pool.py in command(self, dbname, spec, slave_ok, read_preference, codec_options, check, allowable_errors, check_keys, read_concern, write_concern, parse_write_concern_error, collation, session, client, retryable_write, publish_events, user_fields)
    611                            use_op_msg=self.op_msg_enabled,
    612                            unacknowledged=unacknowledged,
--> 613                            user_fields=user_fields)
    614         except OperationFailure:
    615             raise

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/network.py in command(sock, dbname, spec, slave_ok, is_mongos, read_preference, codec_options, session, client, check, allowable_errors, address, check_keys, listeners, max_bson_size, read_concern, parse_write_concern_error, collation, compression_ctx, use_op_msg, unacknowledged, user_fields)
    165                 helpers._check_command_response(
    166                     response_doc, None, allowable_errors,
--> 167                     parse_write_concern_error=parse_write_concern_error)
    168     except Exception as exc:
    169         if publish:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/helpers.py in _check_command_response(response, msg, allowable_errors, parse_write_concern_error)
    157 
    158             msg = msg or "%s"
--> 159             raise OperationFailure(msg % errmsg, code, response)
    160 
    161 

OperationFailure: Index with name: uid_1 already exists with different options

[prjemian]

@jilavsky: Might be best to refactor the tuning to look more like lesson5.

[prjemian]

@jilavsky: Refactored code to use bp.scan() and got same error:

In [2]: RE(preUSAXStune())                                                                                                                                              
I Sat-11:35:24 - Preparing for USAXS mode ... please wait ...
I Sat-11:35:25 - Prepared for USAXS mode
                                                                                                                                                                        
I Sat-11:35:25 - tuning axis: m_stage_r
                                                                                                                                                                        
D Sat-11:35:25 - gain=100000000.0  rate: 1000000.0  max: 950000  converged=[False, False]
D Sat-11:35:25 - gain=10000000.0  rate: 275760.0  max: 950000  converged=[False, False, True, True]
                                                                                                                                                                        
D Sat-11:35:25 - gain=1000000.0  rate: 1000000.0  max: 950000  converged=[False, False]
D Sat-11:35:25 - gain=10000000.0  rate: 274500.0  max: 950000  converged=[False, False, True, True]
                                                                                                                                                                        
D Sat-11:35:26 - gain=10000.0  rate: 1000000.0  max: 950000  converged=[False, False]
D Sat-11:35:26 - gain=10000000.0  rate: 276040.0  max: 950000  converged=[False, False, True, True]
                                                                                                                                                                        
D Sat-11:35:26 - gain=10000.0  rate: 621900.0  max: 950000  converged=[True, True]
D Sat-11:35:26 - gain=10000000.0  rate: 277140.0  max: 950000  converged=[True, True, True, True]
D Sat-11:35:26 - gain=1000000000.0  rate: 3140.0  max: 950000  converged=[True, True, True, True, True, True]
D Sat-11:35:26 - converged: [True, True, True, True, True, True]
I Sat-11:35:26 - Tuning axis m_stage_r, current position is 9.051544
Run aborted
Traceback (most recent call last):
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 1355, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1307, in __call__
    return (yield from plan)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1162, in baseline_wrapper
    return (yield from plan_mutator(plan, insert_baseline))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper
    return (yield from plan2)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper
    return (yield from plan2)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/plans/scans.py", line 123, in preUSAXStune
    yield from tune(md=md)
  File "/home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/plans/axis_tuning.py", line 73, in tune_mr
    yield from _tune_base_(m_stage.r, md=md)
  File "/home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/plans/axis_tuning.py", line 51, in _tune_base_
    yield from axis.tune(md=md)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/apstools/devices.py", line 1036, in tune
    yield from self.tuner.tune(md=md, **kwargs)
  File "/home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/devices/axis_tuning_patches.py", line 174, in tune
    yield from _scan()
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 1045, in dec_inner
    return (yield from plan)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 383, in subs_wrapper
    _unsubscribe()))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 509, in finalize_wrapper
    ret = yield from plan
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 380, in _inner_plan
    return (yield from plan)
  File "/home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/devices/axis_tuning_patches.py", line 171, in _scan
    yield from bp.scan(signal_list, self.axis, start, finish, num, md=_md)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/plans.py", line 1079, in scan
    per_step=per_step, md=_md))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/plans.py", line 973, in scan_nd
    return (yield from inner_scan_nd())
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 1045, in dec_inner
    return (yield from plan)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 952, in stage_wrapper
    return (yield from finalize_wrapper(inner(), unstage_devices()))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 509, in finalize_wrapper
    ret = yield from plan
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 950, in inner
    return (yield from plan)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 1045, in dec_inner
    return (yield from plan)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 317, in run_wrapper
    rs_uid = yield from open_run(md)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 825, in open_run
    return (yield Msg('open_run', **(md or {})))
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator
    msg = plan_stack[-1].throw(exception)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 140, in single_gen
    return (yield msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator
    msg = plan_stack[-1].throw(exception)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 140, in single_gen
    return (yield msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 78, in plan_mutator
    msg = plan_stack[-1].throw(exception)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 140, in single_gen
    return (yield msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 1415, in _run
    new_response = await coro(msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 1600, in _open_run
    new_uid = await current_run.open_run(msg)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/bundlers.py", line 53, in open_run
    await self.emit(DocumentNames.start, doc)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 2262, in emit
    self.emit_sync(name, doc)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 2259, in emit_sync
    self.dispatcher.process(name, doc)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py", line 2282, in process
    exceptions = self.cb_registry.process(name, name.name, doc)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 351, in process
    func(*args, **kwargs)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py", line 432, in __call__
    return mtd(*args, **kwargs)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/v1.py", line 959, in insert
    if self._serializer is None:
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/v1.py", line 189, in _serializer
    self.__serializer = self._catalog._get_serializer()
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/_drivers/mongo_normalized.py", line 351, in _get_serializer
    return Serializer(self._metadatastore_db, self._asset_registry_db)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/suitcase/mongo_normalized/__init__.py", line 48, in __init__
    self._create_indexes()
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/suitcase/mongo_normalized/__init__.py", line 56, in _create_indexes
    self._resource_collection.create_index('uid')
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py", line 1995, in create_index
    self.__create_index(keys, kwargs, session, **cmd_options)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py", line 1894, in __create_index
    session=session)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py", line 250, in _command
    user_fields=user_fields)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/pool.py", line 613, in command
    user_fields=user_fields)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/network.py", line 167, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/helpers.py", line 159, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
Failed to close run <bluesky.bundlers.RunBundler object at 0x7fe6b5b2fd90>. different options
---------------------------------------------------------------------------
OperationFailure                          Traceback (most recent call last)
<ipython-input-2-f92c9110a1e3> in <module>
----> 1 RE(preUSAXStune())

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    795             self._task_fut.add_done_callback(set_blocking_event)
    796 
--> 797         self._resume_task(init_func=_build_task)
    798 
    799         if self._interrupted:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    919                     if (exc is not None
    920                             and not isinstance(exc, _RunEnginePanic)):
--> 921                         raise exc
    922 
    923     def install_suspender(self, suspender):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1488             exit_reason = str(err)
   1489             self.log.exception("Run aborted")
-> 1490             raise err
   1491         finally:
   1492             if not exit_reason:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1353                     else:
   1354                         try:
-> 1355                             msg = self._plan_stack[-1].send(resp)
   1356                         # We have exhausted the top generator
   1357                         except StopIteration:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.ipython/profile_bluesky/startup/instrument/plans/scans.py in preUSAXStune(md)
    121     for axis, tune in tuners.items():
    122         yield from bps.mv(ti_filter_shutter, "open")
--> 123         yield from tune(md=md)
    124         if axis.tuner.tune_ok:
    125             # If we don't wait, the next tune often fails

~/.ipython/profile_bluesky/startup/instrument/plans/axis_tuning.py in tune_mr(md)
     71     md['plan_name'] = "tune_mr"
     72     # print(f"metadata={md}")  # TOO much data to print
---> 73     yield from _tune_base_(m_stage.r, md=md)
     74 
     75 

~/.ipython/profile_bluesky/startup/instrument/plans/axis_tuning.py in _tune_base_(axis, md)
     49     )
     50     yield from autoscale_amplifiers([upd_controls, I0_controls, I00_controls])
---> 51     yield from axis.tune(md=md)
     52     yield from bps.mv(
     53         ti_filter_shutter, "close",

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/apstools/devices.py in tune(self, md, **kwargs)
   1034                 yield from self.pre_tune_method()
   1035 
-> 1036             yield from self.tuner.tune(md=md, **kwargs)
   1037 
   1038             if self.post_tune_method is not None:

~/.ipython/profile_bluesky/startup/instrument/devices/axis_tuning_patches.py in tune(self, width, num, md)
    172             yield from self.peak_analysis(initial_position)
    173 
--> 174         yield from _scan()
    175 
    176     def multi_pass_tune(self, width=None, step_factor=None, 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1043                 plan = gen_func(*inner_args, **inner_kwargs)
   1044                 plan = wrapper(plan, *args, **kwargs)
-> 1045                 return (yield from plan)
   1046             return dec_inner
   1047         return dec

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in subs_wrapper(plan, subs)
    381 
    382     return (yield from finalize_wrapper(_inner_plan(),
--> 383                                         _unsubscribe()))
    384 
    385 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in _inner_plan()
    378     def _inner_plan():
    379         yield from _subscribe()
--> 380         return (yield from plan)
    381 
    382     return (yield from finalize_wrapper(_inner_plan(),

~/.ipython/profile_bluesky/startup/instrument/devices/axis_tuning_patches.py in _scan(md)
    169         @bpp.subs_decorator(self.peaks)
    170         def _scan(md=None):
--> 171             yield from bp.scan(signal_list, self.axis, start, finish, num, md=_md)
    172             yield from self.peak_analysis(initial_position)
    173 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/plans.py in scan(detectors, num, per_step, md, *args)
   1077 
   1078     return (yield from scan_nd(detectors, full_cycler,
-> 1079                                per_step=per_step, md=_md))
   1080 
   1081 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/plans.py in scan_nd(detectors, cycler, per_step, md)
    971             yield from per_step(detectors, step, pos_cache)
    972 
--> 973     return (yield from inner_scan_nd())
    974 
    975 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1043                 plan = gen_func(*inner_args, **inner_kwargs)
   1044                 plan = wrapper(plan, *args, **kwargs)
-> 1045                 return (yield from plan)
   1046             return dec_inner
   1047         return dec

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1043                 plan = gen_func(*inner_args, **inner_kwargs)
   1044                 plan = wrapper(plan, *args, **kwargs)
-> 1045                 return (yield from plan)
   1046             return dec_inner
   1047         return dec

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in run_wrapper(plan, md)
    315         metadata to be passed into the 'open_run' message
    316     """
--> 317     rs_uid = yield from open_run(md)
    318 
    319     def except_plan(e):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/plan_stubs.py in open_run(md)
    823     :func:`bluesky.plans_stubs.close_run`
    824     """
--> 825     return (yield Msg('open_run', **(md or {})))
    826 
    827 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in single_gen(msg)
    138         the input message
    139     '''
--> 140     return (yield msg)
    141 
    142 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in single_gen(msg)
    138         the input message
    139     '''
--> 140     return (yield msg)
    141 
    142 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in single_gen(msg)
    138         the input message
    139     '''
--> 140     return (yield msg)
    141 
    142 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1413                         # exceptions (coming in via throw) can be
   1414                         # raised
-> 1415                         new_response = await coro(msg)
   1416 
   1417                     # special case `CancelledError` and let the outer

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in _open_run(self, msg)
   1598             loop=self.loop)
   1599 
-> 1600         new_uid = await current_run.open_run(msg)
   1601         self._run_start_uids.append(new_uid)
   1602         return new_uid

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/bundlers.py in open_run(self, msg)
     51 
     52         doc = dict(uid=self._run_start_uid, time=ttime.time(), **self._md)
---> 53         await self.emit(DocumentNames.start, doc)
     54         doc_logger.debug("[start] document is emitted (run_uid=%r)", self._run_start_uid,
     55         extra={'doc_name': 'start',

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in emit(self, name, doc)
   2260 
   2261     async def emit(self, name, doc):
-> 2262         self.emit_sync(name, doc)
   2263 
   2264 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in emit_sync(self, name, doc)
   2257         "Process blocking callbacks and schedule non-blocking callbacks."
   2258         schema_validators[name].validate(doc)
-> 2259         self.dispatcher.process(name, doc)
   2260 
   2261     async def emit(self, name, doc):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/run_engine.py in process(self, name, doc)
   2280         doc : dict
   2281         """
-> 2282         exceptions = self.cb_registry.process(name, name.name, doc)
   2283         for exc, traceback in exceptions:
   2284             warn("A %r was raised during the processing of a %s "

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in process(self, sig, *args, **kwargs)
    349             for cid, func in list(self.callbacks[sig].items()):
    350                 try:
--> 351                     func(*args, **kwargs)
    352                 except ReferenceError:
    353                     self._remove_proxy(func)

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/bluesky/utils.py in __call__(self, *args, **kwargs)
    430             mtd = self.func
    431         # invoke the callable and return the result
--> 432         return mtd(*args, **kwargs)
    433 
    434     def __eq__(self, other):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/v1.py in insert(self, name, doc)
    957 
    958     def insert(self, name, doc):
--> 959         if self._serializer is None:
    960             raise RuntimeError("No Serializer was configured for this.")
    961         warnings.warn(

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/v1.py in _serializer(self)
    187             # which is pending deprecation.
    188             if hasattr(self._catalog, '_get_serializer'):
--> 189                 self.__serializer = self._catalog._get_serializer()
    190         return self.__serializer
    191 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/databroker/_drivers/mongo_normalized.py in _get_serializer(self)
    349         "This is used internally by v1.Broker. It may be removed in future."
    350         from suitcase.mongo_normalized import Serializer
--> 351         return Serializer(self._metadatastore_db, self._asset_registry_db)
    352 
    353 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/suitcase/mongo_normalized/__init__.py in __init__(self, metadatastore_db, asset_registry_db)
     46         self._asset_registry_db = assets_db
     47 
---> 48         self._create_indexes()
     49 
     50     def _create_indexes(self):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/suitcase/mongo_normalized/__init__.py in _create_indexes(self)
     54         If the index already exists, this has no effect.
     55         """
---> 56         self._resource_collection.create_index('uid')
     57         self._resource_collection.create_index('resource_id')  # legacy
     58         # TODO: Migrate all Resources to have a RunStart UID, and then make a

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py in create_index(self, keys, session, **kwargs)
   1993         if "maxTimeMS" in kwargs:
   1994             cmd_options["maxTimeMS"] = kwargs.pop("maxTimeMS")
-> 1995         self.__create_index(keys, kwargs, session, **cmd_options)
   1996         return name
   1997 

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py in __create_index(self, keys, index_options, session, **kwargs)
   1892                 codec_options=_UNICODE_REPLACE_CODEC_OPTIONS,
   1893                 write_concern=self._write_concern_for(session),
-> 1894                 session=session)
   1895 
   1896     def create_index(self, keys, session=None, **kwargs):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/collection.py in _command(self, sock_info, command, slave_ok, read_preference, codec_options, check, allowable_errors, read_concern, write_concern, collation, session, retryable_write, user_fields)
    248                 client=self.__database.client,
    249                 retryable_write=retryable_write,
--> 250                 user_fields=user_fields)
    251 
    252     def __create(self, options, collation, session):

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/pool.py in command(self, dbname, spec, slave_ok, read_preference, codec_options, check, allowable_errors, check_keys, read_concern, write_concern, parse_write_concern_error, collation, session, client, retryable_write, publish_events, user_fields)
    611                            use_op_msg=self.op_msg_enabled,
    612                            unacknowledged=unacknowledged,
--> 613                            user_fields=user_fields)
    614         except OperationFailure:
    615             raise

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/network.py in command(sock, dbname, spec, slave_ok, is_mongos, read_preference, codec_options, session, client, check, allowable_errors, address, check_keys, listeners, max_bson_size, read_concern, parse_write_concern_error, collation, compression_ctx, use_op_msg, unacknowledged, user_fields)
    165                 helpers._check_command_response(
    166                     response_doc, None, allowable_errors,
--> 167                     parse_write_concern_error=parse_write_concern_error)
    168     except Exception as exc:
    169         if publish:

/share1/bluesky/conda_envs/bluesky_2020_5/lib/python3.7/site-packages/pymongo/helpers.py in _check_command_response(response, msg, allowable_errors, parse_write_concern_error)
    157 
    158             msg = msg or "%s"
--> 159             raise OperationFailure(msg % errmsg, code, response)
    160 
    161 

OperationFailure: Index with name: uid_1 already exists with different options

In [3]:                                                                                                                                                                 

[prjemian]

@tacaswell wrote on slack:

That looks like a problem on the mongo side, try dropping the mongo indexes and try again

referring to this documentation:

https://docs.mongodb.com/manual/reference/method/db.collection.dropIndexes/ is the very very short version

[prjemian]

db._resource_collection.drop_index('uid')

[prjemian]

might also be ...drop_index("uid_1")

[prjemian]

So, first try RE(bp.count([scaler0])) which results in error as reported:

OperationFailure: Index with name: uid_1 already exists with different options

[prjemian]

then db.v2._resource_collection.drop_index('uid_1') and count again but command line does not return:

In [15]: RE(bp.count([scaler0]))                                                                                                                                             


Transient Scan ID: 5     Time: 2020-06-01 11:15:11
Persistent Unique Scan ID: '5b627e5e-9ac7-4ffa-a016-bb9b3e571064'
New stream: 'baseline'
scaler0 [In progress. No progress bar available.]                                                                                                                            

[prjemian]

because it counts for 312:

In [16]: scaler0.preset_time.get()                                                                                                                                           
Out[16]: 31.0

[prjemian]

Success:

In [17]: scaler0.preset_time.put(0.5)                                                                                                                                        

In [18]: RE(bp.count([scaler0]))                                                                                                                                             


Transient Scan ID: 6     Time: 2020-06-01 11:20:35
Persistent Unique Scan ID: '81ed6b3c-6b0b-4ed3-9d29-bb8b3455d3ff'
New stream: 'baseline'
New stream: 'primary'                                                                                                                                                        
+-----------+------------+------------+------------+------------+------------+------------+------------+
|   seq_num |       time |    seconds |   I0_USAXS |  I00_USAXS |   PD_USAXS |   TR diode |       I000 |
+-----------+------------+------------+------------+------------+------------+------------+------------+
|         1 | 11:20:36.6 |    5000000 |        951 |       1700 |        525 |        318 |          4 |
+-----------+------------+------------+------------+------------+------------+------------+------------+
generator count ['81ed6b3c'] (scan num: 6)
Out[18]: ('81ed6b3c-6b0b-4ed3-9d29-bb8b3455d3ff',)

[prjemian]

@jilavsky: This fixed it. In hind sight, this issue should belong to the USAXS instrument and not apstools.

[jilavsky]

Tested on beamline (with no beam, so it had to fail to find the peak) and confirm, that no error message was generated and scan finished as expected.