Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SPEC filewriter scan numbering when "new" data file exists #128

Closed
prjemian opened this issue May 2, 2019 · 4 comments
Closed

SPEC filewriter scan numbering when "new" data file exists #128

prjemian opened this issue May 2, 2019 · 4 comments
Assignees

Comments

@prjemian
Copy link
Contributor

prjemian commented May 2, 2019

In SpecWriterCallback.newfile(), when kwarg scan_id=None (the default) and the chosen file already exists, the next scan_id should be determined from the existing file. One of these (perhaps the max of all):

  • increment from the last scan number
  • increment from the highest scan number in the file
  • increment from the total number of scans
@prjemian prjemian added this to the milestone-2019-09 milestone May 2, 2019
@prjemian prjemian self-assigned this May 2, 2019
@prjemian
Copy link
Contributor Author

prjemian commented May 2, 2019

Example:

In [8]: fl = !grep "^#S " 05_02_test.dat                                                                                                                               

In [9]: fl                                                                                                                                                             
Out[9]: 
['#S 1  tune_mr()',
 '#S 1  tune_mr()',
 '#S 2  tune_mr()',
 '#S 3  tune_mr()',
 '#S 4  tune_mr()',
 '#S 5  tune_mr()',
 '#S 1  tune_mr()',
 '#S 2  tune_ar()',
 '#S 3  tune_mr()',
 '#S 1  tune_ar()',
 '#S 1  tune_ar()',
 '#S 2  tune_ar()',
 '#S 3  tune_ar()',
 '#S 101  tune_mr()',
 '#S 102  tune_m2rp()',
 '#S 103  tune_ar()',
 '#S 104  tune_a2rp()',
 "#S 105  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 2  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 2  count(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 '#S 1  tune_mr()',
 '#S 1  tune_mr()',
 '#S 1  tune_mr()',
 '#S 108  tune_mr()',
 "#S 109  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 '#S 110  Flyscan(pos_X=60, pos_Y=160, thickness=0, scan_title=blank)']

In [20]: sl = np.array(list(map(int, [f.split()[1] for f in fl])))                                                                                                     

In [21]: sl                                                                                                                                                            
Out[21]: 
array([  1,   1,   2,   3,   4,   5,   1,   2,   3,   1,   1,   2,   3,
       101, 102, 103, 104, 105,   1,   1,   1,   1,   1,   1,   2,   1,
         2,   1,   1,   1,   1,   1,   1,   1,   1,   1, 108, 109, 110])

In [22]: sl.max()                                                                                                                                                      
Out[22]: 110

In [23]: sl[-1]                                                                                                                                                        
Out[23]: 110

In [24]: len(sl)                                                                                                                                                       
Out[24]: 39

In [25]: max(sl[-1], sl.max(), len(sl))                                                                                                                                
Out[25]: 110

Yes, sl[-1] is redundant with sl.max()

@prjemian
Copy link
Contributor Author

prjemian commented May 8, 2019

Should use a package to parse existing SPEC file. Possibilities are silx (ESRF) or spec2nexus (mine)

Here is the relevant silx documentation: http://www.silx.org/doc/silx/latest/modules/io/specfile.html?highlight=spec#module-silx.io.specfile

@prjemian
Copy link
Contributor Author

prjemian commented May 9, 2019

This has bogged down trying to get a databroker document stream into a temporary db backend for testing. We've got the JSON and can read it but the db.insert fails on the first event document.

prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
@prjemian
Copy link
Contributor Author

prjemian commented May 9, 2019

Skipping the databroker for now, we have document streams (from scans with real hardware) ready to use in the unit tests.

prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
prjemian added a commit that referenced this issue May 9, 2019
BUG: SPEC filewriter scan numbering and newfile() when file exists

fixes #125 and #128
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant