wrong format of spec files from sardana

[jkotan]

It looks like spec files from sardana do not adhere to the specifications of the SPEC standard data file format from Standard Data File Format. This causes problems in reading them via spec2nexus prjemian/spec2nexus#122 or nexpy nexpy/nexpy#152

[cpascual]

Before fixing this, I would like to confirm that the official ultimate reference for the spec file syntax is the one linked above. I say so because some years ago, while fixing some other issue in Sardana's spec recorder, we used some reference document for the spec file syntax which I believe it was much more extense and formal than the one in the link... but I cannot find it now among my files or in the web which makes me doubt about my memory... :-\

[rhomspuron]

Hi, you can find the information on the official specpython library. This block is not mandatory:
https://github.com/specpython/filespec/blob/master/specpython/spec_format.rst
For that reason the specpython and pymca can open the file without problems.

On the other hand, the file generated by sardana use the motor label '#O' and position '#P' for the snapshots and it saves them on each scan.

[prjemian]

@vrey01 : Can you comment on this? Which is the definitive reference for the SPEC data file format?

[vrey01]

The spec data format is described in detail in https://certif.com/spec_help/scans.html or, in spec, by typing "help scans". (look for paragraph "File Conventions")

The spec data format gives a list of annotation conventions but it does not specify a "minimal" set of annotations that should be present in a file. For example, in the page referenced above, after the list of standard control lines, there is an example of a "simple data file" containing just "#S, #N and #L" lines. Once this said, as it could be expected, spec itself produces data with much richer annotation that the one in that "simple example", including a header block indicating "filename", "date of creation", "application name", "user name that created the file", "epoch (to precisely time-tag each data point)"...

The tools reading spec data files decide what is the minimal information needed to display data. For example, in the case of the "splot" (standard plot program distributed with spec) it is sufficient to have a "#S" and a "#L" control lines to recognise a file as a valid spec data file and display the scans included in it. In the case of the "scans" unix tool (also distributed with spec) only the "#S" is necessary to properly manage spec files. Eventually each tool will decide how tolerant will be to missing control lines and how it will handle them.
As the spec file format is one for a multi-scan container file I would say that, as scan delimiter, only the "#S" control line is mandatory.

My recommendation for sardana would be to include at least a "file header block", starting with "#F" and including metadata about the file, user, creation time, etc... but.. this is not mandatory!

[vrey01]

Pete, Thanks for pointing that out. I just commented on it in the mailing list For the spec file format conventions you can type “help scans” in spec. There is no mandatory “control line” other than “#S” to separate scans. The “scans” tool, “splot” plot program and “scans.4” c-plot extension, they all support very simple set of control lines. Bixente

…

______________________ Bixente Rey Bakaikoa vrey01@gmail.com +34 696.613.500

El 17 abr 2018, a las 19:54, Pete R Jemian ***@***.***> escribió: @vrey01 <https://github.com/vrey01> : Can you comment on this? Which is the definitive reference for the SPEC data file format? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#735 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFQ4T34iaen-MdLx2lgjqe0CJXz3lgyIks5tpixjgaJpZM4TWnFs>.

[cpascual]

Thanks @vrey01 for the clarification.
Since having the file header is not mandatory, I am marking this issue as "Enhancement" for sardana, but IMHO the original issue should be solved by supporting more generic files in spec2nexus (see prjemian/spec2nexus#122). Do you agreee @prjemian ?

[prjemian]

@vrey01 : thanks for the information. Until this, I have taken the SPEC user manual at its word, while recognizing that there are no rules in SPEC, just a set of macros implementing what is provided from certif.com.

[jkotan]

I've just noticed that there are additionally like #@CHANN, #@MCA_NB, #@DET_0, #@CTIME, #@CALIB, #@ROI in spec_format with respect to scans.

Does it mean that scans is outdated or spec_format is beyond the standard?

[rhomspuron]

We asked to include on filespec new extra-lines: #@MCA_NB, #@DET_0 to save more information related to the 1D detectors.

The other extra-lines are used by PyMCA. AFAIK, spec's scans generates the other extra-lines.

@vrey01: Could you clarify that?

[tiagocoutinho]

I just came across this issue.

I made two PRs:
#1355 - add user information
#1356 - add missing epoch

They may help improve sardana spec recorder

[tiagocoutinho]

Concerning the snapshot feature of sardana:

it is storing each snapshot variable as a separate motor (#O & #o fields)
This is obviously not the best because:

• many snapshot variables are not motors
• many snapshot values are not numbers

I was asked to add more information to the scan header. In my case it corresponds to a sardana environment variable (explicitly, I was asked to avoid storing stuff as motor)

AFAIK, there is no spec field associated with storing general meta information.
I could extend the sardana spec recorder to fit my own purposes by implementing something like #C <variable> <value>

This would probably not break any client. But before I do, does anyone know of any de-facto syntax out there being used for my case?

Thanks in advance

[tiagocoutinho]

Please forget my previous comment. I now see there are already other issues discussing this: #771 and #776.

[vrey01]

Dear Tiago, There is no general metadata information specification in spec format. On the other hand traditionally people have been using the #U lines for that. For example, at the ESRF machine information is stored as lines: #UMI0 #UMI1 #UMI2 Each of them having a particular syntax I would suggest using something like that. As in my opinion the #C lines are often used by macros and users “macros comment or qcomment” in spec to write any annotation they like. That may eventually conflict with a more structured information. I would leave #C for totally free comments that do not need to be parsed. #U for information with a structure that can be later parsed with confidence. Examples for metadata could be: #UVAR <varname> <value> or #UMETA var1=value1 var2=value2 var3=value3 Feel free to propose or use Best Bixente

…

______________________ Bixente Rey Bakaikoa vrey01@gmail.com +34 696.613.500

El 17 jun 2020, a las 18:22, Tiago Coutinho ***@***.***> escribió: Concerning the snapshot feature of sardana: it is storing each snapshot variable as a separate motor (#O & #o fields) This is obviously not the best because: many snapshot variables are not motors many snapshot values are not numbers I was asked to add more information to the scan header. In my case it corresponds to a sardana environment variable (explicitly, I was asked to avoid storing stuff as motor) AFAIK, there is no spec field associated with storing general meta information. I could extend the sardana spec recorder to fit my own purposes by implementing something like #C <variable> <value> This would probably not break any client. But before I do, does anyone know of any de-facto syntax out there being used for my case? Thanks in advance — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#735 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKDQT3X7YM4GPBSHGMHFH3RXDUUNANCNFSM4E22OFWA>.

[tiagocoutinho]

Cool, thanks @vrey01.
That is valuable information I was not aware of.

[guifrecuni]

Yes! it is really a good contribution to think about.
Both #UVAR and #UMETA proposed options look good to me.
Also, these macros for "adding comments" may be useful for Sardana users.

Thanks Bixente!

[tiagocoutinho]

I see we have a addCustomData() that produces #C <key> : <value>.

Should we:

• change it to produce #UVAR <varname> <value> or
• add another method?

[cpascual]

I see we have a addCustomData() that produces #C :

Note that addCustomData is called by the DataHandler on all recorders if the macro defines custom data. If you create a different one, it won't be accessible in this way.

I personally would add a kwarg to the SpecFileRecorder.addCustomData to specify a format string and I would set it as an option in sardanacustomsettings (so that anyone looking for bck-compat can recover the previous behaviour):

class SPEC_FileRecorder(BaseFileRecorder):
    ...
    def _addCustomData(self, value, name, spec_custom_fmt=None, **kwargs):
        if spec_custom_fmt is None:
            spec_custom_fmt = getattr(sardanacustomsettings, "SPEC_CUSTOM_DATA_FORMAT", "#UVAR {name} {value}")
       ...
       self.fd.write(spec_custom_fmt.format(value=v, name=name))

And in sardanacustomsettings, SPEC_CUSTOM_DATA_FORMAT would be defined with "#UVAR {name} {value}" but documented with a suggestion of "#C {name} : {value}" for bck-compat

[dschick]

at bessy we also use the #U tag but instead of having a key: value syntax we do like

#u1 key1 key2 key3
#U1 value1 value2 value3

Is it also possible to add the custom data to hdf5 files?

[cpascual]

Is it also possible to add the custom data to hdf5 files?

Yes. It is a recorder-agnostic API (although you can pass recorder-specific arguments which will be ignored by the other recorders) and the h5FileRecorder supports it