-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Specwriter problem #440
Comments
Thanks for the report. |
Found it: >>> from databroker import catalog
>>> cat = catalog["4id_polar"]
>>> run = cat["624e776a"]
>>> run
BlueskyRun
uid='624e776a-a914-4a74-8841-babf1591fb29'
exit_status='success'
2020-09-15 19:37:32.469 -- 2020-09-15 19:42:19.791
Streams:
* primary
* baseline |
Use databroker v0 interface: with the specwriter callback Note, in the polar account, your databroker configuration is called import databroker
from apstools.filewriters import SpecWriterCallback
specwriter = SpecWriterCallback()
db = databroker.Broker.named('4id_polar')
for key, doc in db.get_documents(db["624e776a"]):
specwriter.receiver(key, doc)
print("Look at SPEC data file: "+specwriter.spec_filename) In the ipython session:
Here is the content of the SPEC file:
|
This file can be read by pymca and content is plotted. @gfabbris Did some other scan (uid) produce the same problem? I'm not seeing it with |
Indeed. Plot shows more than the 7 points described in the upper, left panel. Can you post the header block and the specific scan block from that file here? Use three ` characters on a line before and after each block. |
Here is the header of that scan:
|
it's interesting that the data is correctly saved into the file, but pymca can't read it for some reason. Also, this issue happens to every
|
If it is true that the |
Yeah, that's probably the issue, I tested in a text editor and It does break into lines. It's interesting that your |
Note, by the way, that the qx_scan parameters are not saved in the start document anymore. They are now a device that is read in the baseline stream. |
Current version is 1.3.8. I'm working on 1.3.9 to be released before the
end of the month.
…On Mon, Nov 16, 2020, 1:37 PM Gilberto Fabbris ***@***.***> wrote:
Yeah, that's probably the issue, I tested in a text editor and It does
break into lines. It's interesting that your SpecWriterCallback works,
but the one at the beamline doesn't, was it updated recently? We are using
apstools 1.3.4.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#440 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMBRMEQF4O37E3EHIK3SQF5RFANCNFSM4SNPT2KA>
.
|
You were running 1.3.4 but there have been no changes to the SPEC writer since that version. In [1]: from databroker import catalog
In [2]: run = catalog["4id_polar"]["624e776a"]
In [4]: run._run_start_doc["versions"]["apstools"]
Out[4]: '1.3.4' Same version as you have today, by the way: In [5]: run = catalog["4id_polar"][-1]
In [6]: run
Out[6]:
BlueskyRun
uid='637992e6-3d7b-4e4a-9dd4-ed82d881e89e'
exit_status='success'
2020-11-19 11:15:48.774 -- 2020-11-19 11:16:20.291
Streams:
* primary
* baseline
In [7]: run._run_start_doc["versions"]["apstools"]
Out[7]: '1.3.4' |
Going through the same steps with the installation as-is on the polar account: ipython stepslift% bash
bash-4.2$ export CONDA_ACTIVATE=/APSshare/miniconda/x86_64/bin/activate
bash-4.2$ export CONDA_ENVIRONMENT=bluesky_2020_9
bash-4.2$
bash-4.2$ source ${CONDA_ACTIVATE} ${CONDA_ENVIRONMENT}
(bluesky_2020_9) bash-4.2$ ipython
Python 3.8.2 (default, Mar 26 2020, 15:53:00)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from databroker import catalog
In [4]: cat = catalog["mongodb_config"]
In [5]: db = cat.v1
In [6]: h = db["624e776a"]
In [7]: h.start["versions"]["apstools"]
Out[7]: '1.3.4'
In [8]: import apstools
In [9]: apstools.__version__
Out[9]: '1.3.4'
In [10]: from apstools.filewriters import SpecWriterCallback
In [11]: specwriter = SpecWriterCallback()
In [12]: !pwd
/home/beams/POLAR
In [13]: for key, doc in db.get_documents(db["624e776a"]):
...: specwriter.receiver(key, doc)
...:
In [14]: print("Look at SPEC data file: "+specwriter.spec_filename)
Look at SPEC data file: 20201119-150646.dat
In [15]: !code -n 20201119-150646.dat Looking at the SPEC file these steps produced just now, the first line of the scan is all one line, not broken into several. |
@gfabbris : Can you reproduce the error with a new scan? |
This issue came up when I had I wonder if this has anything to do with having a plan with some "long" argument. |
Ok, I could reproduce the problem, and also I think I understand what's causing it. It seems to me that the problem comes up if one of the argument is a Using my computer, I imported
Then I ran:
This is the spec file it generated.
And, since there is a line break in the second scan, pymca cannot read it: |
Thanks for the details!
…On Sat, Nov 21, 2020, 5:42 PM Gilberto Fabbris ***@***.***> wrote:
Ok, I could reproduce the problem, and also I think I understand what's
causing it. It seems to me that the problem comes up if one of the argument
is a numpy.array. Here it is:
Using my computer, I imported ophyd.sim.motor and ophyd.sim.det. I'm
using apstools 1.2.6, and defined the plan below.
In [66]: def new_plan(detector,positioner,start,end,points,longvariable):
...:
...: positioner = [positioner]
...: _md = {'detectors': [det.name for det in detector],
...: 'motors': [pos.name for pos in positioner],
...: 'num_points': points,
...: 'num_intervals': points - 1,
...: 'plan_args': {'detectors': list(map(repr, detector)),
...: 'motor': list(map(repr,positioner)),
...: 'start': start,
...: 'end': end,
...: 'points': points,
...: 'longvariable':longvariable},
...: 'plan_name': 'new_plan',
...: 'hints': {},
...: }
...:
...: steps = np.linspace(start,end,points)
...:
...: @run_decorator(md=_md)
...: def _innerscan():
...: for step in steps:
...: grp = short_uid('set')
...: yield Msg('checkpoint')
...: yield from mv(positioner[0],step,group=grp)
...: yield from trigger_and_read(detector + positioner)
...:
...: yield from _innerscan()
Then I ran:
In [79]: short_var = {'short':np.linspace(0,10,2)}
In [80]: RE(new_plan([det],motor,-1,1,10,short_var))
Out[80]: ('15559cd9-12a6-4fb4-a49c-d7e30d8a171c',)
In [81]: long_var = {'long':np.linspace(0,10,100)}
In [82]: RE(new_plan([det],motor,-1,1,10,long_var))
Out[82]: ('66b3190b-9097-4278-907c-141e619859f2',)
In [83]: long_var = {'long':list(np.linspace(0,10,100))}
In [84]: RE(new_plan([det],motor,-1,1,10,long_var))
Out[84]: ('aa538833-776f-43d2-ae5c-2f6429e2eaa8',)
This is the spec file it generated.
#S 17 new_plan(detectors=['det'], motor=['motor'], start=-1, end=1, points=10, longvariable={'short': array([ 0., 10.])})
#D Sat Nov 21 17:38:31 2020
#C Sat Nov 21 17:38:31 2020. plan_type = generator
#C Sat Nov 21 17:38:31 2020. uid = 15559cd9-12a6-4fb4-a49c-d7e30d8a171c
#MD uid = 15559cd9-12a6-4fb4-a49c-d7e30d8a171c
#MD detectors = ['det']
#MD motors = ['motor']
#MD num_intervals = 9
#MD num_points = 10
#MD versions = {'ophyd': '1.5.2', 'bluesky': '1.6.4'}
#P0
#N 5
#L motor Epoch_float Epoch motor_setpoint det
-1.0 0.015300989151000977 0 -1.0 0.6065306597126334
-0.7777777777777778 0.02045917510986328 0 -0.7777777777777778 0.7389912962803088
-0.5555555555555556 0.024955034255981445 0 -0.5555555555555556 0.8569968914352789
-0.33333333333333337 0.030132055282592773 0 -0.33333333333333337 0.9459594689067654
-0.11111111111111116 0.03316497802734375 0 -0.11111111111111116 0.9938461733264412
0.11111111111111116 0.03566908836364746 0 0.11111111111111116 0.9938461733264412
0.33333333333333326 0.038001060485839844 0 0.33333333333333326 0.9459594689067655
0.5555555555555554 0.0407719612121582 0 0.5555555555555554 0.856996891435279
0.7777777777777777 0.04359698295593262 0 0.7777777777777777 0.7389912962803089
1.0 0.05047798156738281 0 1.0 0.6065306597126334
#C Sat Nov 21 17:38:31 2020. num_events_primary = 10
#C Sat Nov 21 17:38:31 2020. exit_status = success
#S 18 new_plan(detectors=['det'], motor=['motor'], start=-1, end=1, points=10, longvariable={'long': array([ 0. , 0.1010101 , 0.2020202 , 0.3030303 , 0.4040404 ,
0.50505051, 0.60606061, 0.70707071, 0.80808081, 0.90909091,
1.01010101, 1.11111111, 1.21212121, 1.31313131, 1.41414141,
1.51515152, 1.61616162, 1.71717172, 1.81818182, 1.91919192,
2.02020202, 2.12121212, 2.22222222, 2.32323232, 2.42424242,
2.52525253, 2.62626263, 2.72727273, 2.82828283, 2.92929293,
3.03030303, 3.13131313, 3.23232323, 3.33333333, 3.43434343,
3.53535354, 3.63636364, 3.73737374, 3.83838384, 3.93939394,
4.04040404, 4.14141414, 4.24242424, 4.34343434, 4.44444444,
4.54545455, 4.64646465, 4.74747475, 4.84848485, 4.94949495,
5.05050505, 5.15151515, 5.25252525, 5.35353535, 5.45454545,
5.55555556, 5.65656566, 5.75757576, 5.85858586, 5.95959596,
6.06060606, 6.16161616, 6.26262626, 6.36363636, 6.46464646,
6.56565657, 6.66666667, 6.76767677, 6.86868687, 6.96969697,
7.07070707, 7.17171717, 7.27272727, 7.37373737, 7.47474747,
7.57575758, 7.67676768, 7.77777778, 7.87878788, 7.97979798,
8.08080808, 8.18181818, 8.28282828, 8.38383838, 8.48484848,
8.58585859, 8.68686869, 8.78787879, 8.88888889, 8.98989899,
9.09090909, 9.19191919, 9.29292929, 9.39393939, 9.49494949,
9.5959596 , 9.6969697 , 9.7979798 , 9.8989899 , 10. ])})
#D Sat Nov 21 17:38:39 2020
#C Sat Nov 21 17:38:39 2020. plan_type = generator
#C Sat Nov 21 17:38:39 2020. uid = 66b3190b-9097-4278-907c-141e619859f2
#MD uid = 66b3190b-9097-4278-907c-141e619859f2
#MD detectors = ['det']
#MD motors = ['motor']
#MD num_intervals = 9
#MD num_points = 10
#MD versions = {'ophyd': '1.5.2', 'bluesky': '1.6.4'}
#P0
#N 5
#L motor Epoch_float Epoch motor_setpoint det
-1.0 0.01544499397277832 0 -1.0 0.6065306597126334
-0.7777777777777778 0.020952701568603516 0 -0.7777777777777778 0.7389912962803088
-0.5555555555555556 0.025393962860107422 0 -0.5555555555555556 0.8569968914352789
-0.33333333333333337 0.02995777130126953 0 -0.33333333333333337 0.9459594689067654
-0.11111111111111116 0.03400087356567383 0 -0.11111111111111116 0.9938461733264412
0.11111111111111116 0.036563873291015625 0 0.11111111111111116 0.9938461733264412
0.33333333333333326 0.03850078582763672 0 0.33333333333333326 0.9459594689067655
0.5555555555555554 0.04083895683288574 0 0.5555555555555554 0.856996891435279
0.7777777777777777 0.04688382148742676 0 0.7777777777777777 0.7389912962803089
1.0 0.04942893981933594 0 1.0 0.6065306597126334
#C Sat Nov 21 17:38:39 2020. num_events_primary = 10
#C Sat Nov 21 17:38:39 2020. exit_status = success
#S 19 new_plan(detectors=['det'], motor=['motor'], start=-1, end=1, points=10, longvariable={'long': [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0]})
#D Sat Nov 21 17:38:45 2020
#C Sat Nov 21 17:38:45 2020. plan_type = generator
#C Sat Nov 21 17:38:45 2020. uid = aa538833-776f-43d2-ae5c-2f6429e2eaa8
#MD uid = aa538833-776f-43d2-ae5c-2f6429e2eaa8
#MD detectors = ['det']
#MD motors = ['motor']
#MD num_intervals = 9
#MD num_points = 10
#MD versions = {'ophyd': '1.5.2', 'bluesky': '1.6.4'}
#P0
#N 5
#L motor Epoch_float Epoch motor_setpoint det
-1.0 0.014036893844604492 0 -1.0 0.6065306597126334
-0.7777777777777778 0.018890857696533203 0 -0.7777777777777778 0.7389912962803088
-0.5555555555555556 0.02501082420349121 0 -0.5555555555555556 0.8569968914352789
-0.33333333333333337 0.03194999694824219 0 -0.33333333333333337 0.9459594689067654
-0.11111111111111116 0.03591489791870117 0 -0.11111111111111116 0.9938461733264412
0.11111111111111116 0.03852105140686035 0 0.11111111111111116 0.9938461733264412
0.33333333333333326 0.040413856506347656 0 0.33333333333333326 0.9459594689067655
0.5555555555555554 0.04235386848449707 0 0.5555555555555554 0.856996891435279
0.7777777777777777 0.04468393325805664 0 0.7777777777777777 0.7389912962803089
1.0 0.05132889747619629 0 1.0 0.6065306597126334
#C Sat Nov 21 17:38:45 2020. num_events_primary = 10
#C Sat Nov 21 17:38:45 2020. exit_status = success
And, since there is a line break in the second scan, pymca cannot read it:
[image: Screen Shot 2020-11-21 at 5 40 14 PM]
<https://user-images.githubusercontent.com/10960331/99890026-a42e3780-2c20-11eb-8828-455a335484f5.png>
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#440 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMF2XNDMGAAJQPP4R5TSRBF4PANCNFSM4SNPT2KA>
.
|
bash command to extract the desired run:
but this failed to pack, reporting this error:
|
Aha. Not at 8ID, it's Polar 4-ID. Try again from my computer (
which showed this output to the console:
|
tar that directory for work on this issue
|
Issue #479 will follow from here for extracting this Documentation is here: https://blueskyproject.io/databroker-pack/usage.html#unpacking-a-packed-catalog |
Ready for business! (P.S. Don't forget to delete the configuration file at |
Hint, the search path can be learned from: In [15]: databroker.catalog_search_path()
Out[15]:
('/home/prjemian/.local/share/intake',
'/home/prjemian/Apps/anaconda/envs/bluesky_2021_1/share/intake') |
Also, for unit testing, best to untar that |
Again, without the unpack step:
This way, no new catalog is added to the catalog_search_path. |
Note this important comment from the documentation:
In our case, we packed it into |
@gfabbris - I have reproduced the problem in a unit test and have identified the one line of code that enables this problem (it makes an assumption which is too simplistic): apstools/apstools/filewriters.py Line 76 in 5041417
A fix will involve a refactoring of the apstools/apstools/filewriters.py Line 66 in 5041417
|
Actually, this line is the problem (with the expansion of apstools/apstools/filewriters.py Line 98 in 5041417
Adding a new function to parse |
@prjemian: The specwriter didn't work for scan uid 624e776a. It looks like it's a problem related with how we move the undulator energy in moveE. If I move only the monochromator this issue doesn't happen.
The text was updated successfully, but these errors were encountered: