Specwriter problem #440
Comments
|
Thanks for the report. |
|
Found it: >>> from databroker import catalog
>>> cat = catalog["4id_polar"]
>>> run = cat["624e776a"]
>>> run
BlueskyRun
uid='624e776a-a914-4a74-8841-babf1591fb29'
exit_status='success'
2020-09-15 19:37:32.469 -- 2020-09-15 19:42:19.791
Streams:
* primary
* baseline |
|
Use databroker v0 interface: with the specwriter callback Note, in the polar account, your databroker configuration is called import databroker
from apstools.filewriters import SpecWriterCallback
specwriter = SpecWriterCallback()
db = databroker.Broker.named('4id_polar')
for key, doc in db.get_documents(db["624e776a"]):
specwriter.receiver(key, doc)
print("Look at SPEC data file: "+specwriter.spec_filename)In the ipython session: Here is the content of the SPEC file: |
|
This file can be read by pymca and content is plotted. @gfabbris Did some other scan (uid) produce the same problem? I'm not seeing it with |
|
This is odd. This how that scan appears in the spec file generated during the scan:
|
|
Indeed. Plot shows more than the 7 points described in the upper, left panel. Can you post the header block and the specific scan block from that file here? Use three ` characters on a line before and after each block. |
|
Here is the header of that scan: |
|
it's interesting that the data is correctly saved into the file, but pymca can't read it for some reason. Also, this issue happens to every |
|
If it is true that the |
|
Yeah, that's probably the issue, I tested in a text editor and It does break into lines. It's interesting that your |
|
Note, by the way, that the qx_scan parameters are not saved in the start document anymore. They are now a device that is read in the baseline stream. |
|
Current version is 1.3.8. I'm working on 1.3.9 to be released before the
end of the month.
…On Mon, Nov 16, 2020, 1:37 PM Gilberto Fabbris ***@***.***> wrote:
Yeah, that's probably the issue, I tested in a text editor and It does
break into lines. It's interesting that your SpecWriterCallback works,
but the one at the beamline doesn't, was it updated recently? We are using
apstools 1.3.4.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#440 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMBRMEQF4O37E3EHIK3SQF5RFANCNFSM4SNPT2KA>
.
|
|
You were running 1.3.4 but there have been no changes to the SPEC writer since that version. In [1]: from databroker import catalog
In [2]: run = catalog["4id_polar"]["624e776a"]
In [4]: run._run_start_doc["versions"]["apstools"]
Out[4]: '1.3.4'Same version as you have today, by the way: In [5]: run = catalog["4id_polar"][-1]
In [6]: run
Out[6]:
BlueskyRun
uid='637992e6-3d7b-4e4a-9dd4-ed82d881e89e'
exit_status='success'
2020-11-19 11:15:48.774 -- 2020-11-19 11:16:20.291
Streams:
* primary
* baseline
In [7]: run._run_start_doc["versions"]["apstools"]
Out[7]: '1.3.4' |
|
Going through the same steps with the installation as-is on the polar account: ipython stepslift% bash
bash-4.2$ export CONDA_ACTIVATE=/APSshare/miniconda/x86_64/bin/activate
bash-4.2$ export CONDA_ENVIRONMENT=bluesky_2020_9
bash-4.2$
bash-4.2$ source ${CONDA_ACTIVATE} ${CONDA_ENVIRONMENT}
(bluesky_2020_9) bash-4.2$ ipython
Python 3.8.2 (default, Mar 26 2020, 15:53:00)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from databroker import catalog
In [4]: cat = catalog["mongodb_config"]
In [5]: db = cat.v1
In [6]: h = db["624e776a"]
In [7]: h.start["versions"]["apstools"]
Out[7]: '1.3.4'
In [8]: import apstools
In [9]: apstools.__version__
Out[9]: '1.3.4'
In [10]: from apstools.filewriters import SpecWriterCallback
In [11]: specwriter = SpecWriterCallback()
In [12]: !pwd
/home/beams/POLAR
In [13]: for key, doc in db.get_documents(db["624e776a"]):
...: specwriter.receiver(key, doc)
...:
In [14]: print("Look at SPEC data file: "+specwriter.spec_filename)
Look at SPEC data file: 20201119-150646.dat
In [15]: !code -n 20201119-150646.dat Looking at the SPEC file these steps produced just now, the first line of the scan is all one line, not broken into several.
|
|
@gfabbris : Can you reproduce the error with a new scan? |
|
This issue came up when I had I wonder if this has anything to do with having a plan with some "long" argument. |
|
Ok, I could reproduce the problem, and also I think I understand what's causing it. It seems to me that the problem comes up if one of the argument is a Using my computer, I imported Then I ran: This is the spec file it generated. And, since there is a line break in the second scan, pymca cannot read it:
|
|
Thanks for the details!
…On Sat, Nov 21, 2020, 5:42 PM Gilberto Fabbris ***@***.***> wrote:
Ok, I could reproduce the problem, and also I think I understand what's
causing it. It seems to me that the problem comes up if one of the argument
is a numpy.array. Here it is:
Using my computer, I imported ophyd.sim.motor and ophyd.sim.det. I'm
using apstools 1.2.6, and defined the plan below.
In [66]: def new_plan(detector,positioner,start,end,points,longvariable):
...:
...: positioner = [positioner]
...: _md = {'detectors': [det.name for det in detector],
...: 'motors': [pos.name for pos in positioner],
...: 'num_points': points,
...: 'num_intervals': points - 1,
...: 'plan_args': {'detectors': list(map(repr, detector)),
...: 'motor': list(map(repr,positioner)),
...: 'start': start,
...: 'end': end,
...: 'points': points,
...: 'longvariable':longvariable},
...: 'plan_name': 'new_plan',
...: 'hints': {},
...: }
...:
...: steps = np.linspace(start,end,points)
...:
...: @run_decorator(md=_md)
...: def _innerscan():
...: for step in steps:
...: grp = short_uid('set')
...: yield Msg('checkpoint')
...: yield from mv(positioner[0],step,group=grp)
...: yield from trigger_and_read(detector + positioner)
...:
...: yield from _innerscan()
Then I ran:
In [79]: short_var = {'short':np.linspace(0,10,2)}
In [80]: RE(new_plan([det],motor,-1,1,10,short_var))
Out[80]: ('15559cd9-12a6-4fb4-a49c-d7e30d8a171c',)
In [81]: long_var = {'long':np.linspace(0,10,100)}
In [82]: RE(new_plan([det],motor,-1,1,10,long_var))
Out[82]: ('66b3190b-9097-4278-907c-141e619859f2',)
In [83]: long_var = {'long':list(np.linspace(0,10,100))}
In [84]: RE(new_plan([det],motor,-1,1,10,long_var))
Out[84]: ('aa538833-776f-43d2-ae5c-2f6429e2eaa8',)
This is the spec file it generated.
#S 17 new_plan(detectors=['det'], motor=['motor'], start=-1, end=1, points=10, longvariable={'short': array([ 0., 10.])})
#D Sat Nov 21 17:38:31 2020
#C Sat Nov 21 17:38:31 2020. plan_type = generator
#C Sat Nov 21 17:38:31 2020. uid = 15559cd9-12a6-4fb4-a49c-d7e30d8a171c
#MD uid = 15559cd9-12a6-4fb4-a49c-d7e30d8a171c
#MD detectors = ['det']
#MD motors = ['motor']
#MD num_intervals = 9
#MD num_points = 10
#MD versions = {'ophyd': '1.5.2', 'bluesky': '1.6.4'}
#P0
#N 5
#L motor Epoch_float Epoch motor_setpoint det
-1.0 0.015300989151000977 0 -1.0 0.6065306597126334
-0.7777777777777778 0.02045917510986328 0 -0.7777777777777778 0.7389912962803088
-0.5555555555555556 0.024955034255981445 0 -0.5555555555555556 0.8569968914352789
-0.33333333333333337 0.030132055282592773 0 -0.33333333333333337 0.9459594689067654
-0.11111111111111116 0.03316497802734375 0 -0.11111111111111116 0.9938461733264412
0.11111111111111116 0.03566908836364746 0 0.11111111111111116 0.9938461733264412
0.33333333333333326 0.038001060485839844 0 0.33333333333333326 0.9459594689067655
0.5555555555555554 0.0407719612121582 0 0.5555555555555554 0.856996891435279
0.7777777777777777 0.04359698295593262 0 0.7777777777777777 0.7389912962803089
1.0 0.05047798156738281 0 1.0 0.6065306597126334
#C Sat Nov 21 17:38:31 2020. num_events_primary = 10
#C Sat Nov 21 17:38:31 2020. exit_status = success
#S 18 new_plan(detectors=['det'], motor=['motor'], start=-1, end=1, points=10, longvariable={'long': array([ 0. , 0.1010101 , 0.2020202 , 0.3030303 , 0.4040404 ,
0.50505051, 0.60606061, 0.70707071, 0.80808081, 0.90909091,
1.01010101, 1.11111111, 1.21212121, 1.31313131, 1.41414141,
1.51515152, 1.61616162, 1.71717172, 1.81818182, 1.91919192,
2.02020202, 2.12121212, 2.22222222, 2.32323232, 2.42424242,
2.52525253, 2.62626263, 2.72727273, 2.82828283, 2.92929293,
3.03030303, 3.13131313, 3.23232323, 3.33333333, 3.43434343,
3.53535354, 3.63636364, 3.73737374, 3.83838384, 3.93939394,
4.04040404, 4.14141414, 4.24242424, 4.34343434, 4.44444444,
4.54545455, 4.64646465, 4.74747475, 4.84848485, 4.94949495,
5.05050505, 5.15151515, 5.25252525, 5.35353535, 5.45454545,
5.55555556, 5.65656566, 5.75757576, 5.85858586, 5.95959596,
6.06060606, 6.16161616, 6.26262626, 6.36363636, 6.46464646,
6.56565657, 6.66666667, 6.76767677, 6.86868687, 6.96969697,
7.07070707, 7.17171717, 7.27272727, 7.37373737, 7.47474747,
7.57575758, 7.67676768, 7.77777778, 7.87878788, 7.97979798,
8.08080808, 8.18181818, 8.28282828, 8.38383838, 8.48484848,
8.58585859, 8.68686869, 8.78787879, 8.88888889, 8.98989899,
9.09090909, 9.19191919, 9.29292929, 9.39393939, 9.49494949,
9.5959596 , 9.6969697 , 9.7979798 , 9.8989899 , 10. ])})
#D Sat Nov 21 17:38:39 2020
#C Sat Nov 21 17:38:39 2020. plan_type = generator
#C Sat Nov 21 17:38:39 2020. uid = 66b3190b-9097-4278-907c-141e619859f2
#MD uid = 66b3190b-9097-4278-907c-141e619859f2
#MD detectors = ['det']
#MD motors = ['motor']
#MD num_intervals = 9
#MD num_points = 10
#MD versions = {'ophyd': '1.5.2', 'bluesky': '1.6.4'}
#P0
#N 5
#L motor Epoch_float Epoch motor_setpoint det
-1.0 0.01544499397277832 0 -1.0 0.6065306597126334
-0.7777777777777778 0.020952701568603516 0 -0.7777777777777778 0.7389912962803088
-0.5555555555555556 0.025393962860107422 0 -0.5555555555555556 0.8569968914352789
-0.33333333333333337 0.02995777130126953 0 -0.33333333333333337 0.9459594689067654
-0.11111111111111116 0.03400087356567383 0 -0.11111111111111116 0.9938461733264412
0.11111111111111116 0.036563873291015625 0 0.11111111111111116 0.9938461733264412
0.33333333333333326 0.03850078582763672 0 0.33333333333333326 0.9459594689067655
0.5555555555555554 0.04083895683288574 0 0.5555555555555554 0.856996891435279
0.7777777777777777 0.04688382148742676 0 0.7777777777777777 0.7389912962803089
1.0 0.04942893981933594 0 1.0 0.6065306597126334
#C Sat Nov 21 17:38:39 2020. num_events_primary = 10
#C Sat Nov 21 17:38:39 2020. exit_status = success
#S 19 new_plan(detectors=['det'], motor=['motor'], start=-1, end=1, points=10, longvariable={'long': [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0]})
#D Sat Nov 21 17:38:45 2020
#C Sat Nov 21 17:38:45 2020. plan_type = generator
#C Sat Nov 21 17:38:45 2020. uid = aa538833-776f-43d2-ae5c-2f6429e2eaa8
#MD uid = aa538833-776f-43d2-ae5c-2f6429e2eaa8
#MD detectors = ['det']
#MD motors = ['motor']
#MD num_intervals = 9
#MD num_points = 10
#MD versions = {'ophyd': '1.5.2', 'bluesky': '1.6.4'}
#P0
#N 5
#L motor Epoch_float Epoch motor_setpoint det
-1.0 0.014036893844604492 0 -1.0 0.6065306597126334
-0.7777777777777778 0.018890857696533203 0 -0.7777777777777778 0.7389912962803088
-0.5555555555555556 0.02501082420349121 0 -0.5555555555555556 0.8569968914352789
-0.33333333333333337 0.03194999694824219 0 -0.33333333333333337 0.9459594689067654
-0.11111111111111116 0.03591489791870117 0 -0.11111111111111116 0.9938461733264412
0.11111111111111116 0.03852105140686035 0 0.11111111111111116 0.9938461733264412
0.33333333333333326 0.040413856506347656 0 0.33333333333333326 0.9459594689067655
0.5555555555555554 0.04235386848449707 0 0.5555555555555554 0.856996891435279
0.7777777777777777 0.04468393325805664 0 0.7777777777777777 0.7389912962803089
1.0 0.05132889747619629 0 1.0 0.6065306597126334
#C Sat Nov 21 17:38:45 2020. num_events_primary = 10
#C Sat Nov 21 17:38:45 2020. exit_status = success
And, since there is a line break in the second scan, pymca cannot read it:
[image: Screen Shot 2020-11-21 at 5 40 14 PM]
<https://user-images.githubusercontent.com/10960331/99890026-a42e3780-2c20-11eb-8828-455a335484f5.png>
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#440 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMF2XNDMGAAJQPP4R5TSRBF4PANCNFSM4SNPT2KA>
.
|
|
bash command to extract the desired run: but this failed to pack, reporting this error: |
|
Aha. Not at 8ID, it's Polar 4-ID. Try again from my computer ( which showed this output to the console: |
|
tar that directory for work on this issue |
|
Issue #479 will follow from here for extracting this Documentation is here: https://blueskyproject.io/databroker-pack/usage.html#unpacking-a-packed-catalog |
Ready for business! (P.S. Don't forget to delete the configuration file at |
|
Hint, the search path can be learned from: In [15]: databroker.catalog_search_path()
Out[15]:
('/home/prjemian/.local/share/intake',
'/home/prjemian/Apps/anaconda/envs/bluesky_2021_1/share/intake') |
|
Also, for unit testing, best to untar that |
|
Again, without the unpack step: This way, no new catalog is added to the catalog_search_path. |
|
Note this important comment from the documentation:
In our case, we packed it into |
|
@gfabbris - I have reproduced the problem in a unit test and have identified the one line of code that enables this problem (it makes an assumption which is too simplistic): apstools/apstools/filewriters.py Line 76 in 5041417 A fix will involve a refactoring of the apstools/apstools/filewriters.py Line 66 in 5041417 |
|
Actually, this line is the problem (with the expansion of apstools/apstools/filewriters.py Line 98 in 5041417 Adding a new function to parse |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@prjemian: The specwriter didn't work for scan uid 624e776a. It looks like it's a problem related with how we move the undulator energy in moveE. If I move only the monochromator this issue doesn't happen.
The text was updated successfully, but these errors were encountered: