Add a templating system to the NXWriter callback

[prjemian]

Thanks to @danielballan for the prompt on bluesky's Mattermost discussion site, it could be easier for users to write files compliant with NeXus base classes and application definitions through a templating (mapping, some other names, ...) system.

Under the assumption that all the information content from a bluesky run has been written somewhere under the NXentry group by the NXWriter, a template item would map that content (via NeXus/HDF5 link) to the group or field where it is needed to satisfy the NeXus structure. The somewhere locations are under /entry/instrument/bluesky or in specific known locations by the existing NXWriter code.

For example, the NXentry group expects a field called entry_identifier. This is linked by existing code: entry_identifier --> /entry/instrument/bluesky/metadata/run_start_uid

[prjemian]

A set of templates specific for the run could be passed to the NXWriter through the run's metadata. The templates would be composed as a Python dictionary. To use the run metadata, it is necessary to write them as a string. JSON is good for this.

md = {
    "title": "Develop NeXus/HDF5 template support", 
    nxwriter.template_key: json.dumps(template),
}

Here nxwriter.template_key is the metadata key where the templates will be found by the NXWriter. (such as nxwriter.template_key = "nxwriter_template")

The template would be handled just before this code returns:

apstools/apstools/callbacks/nexus_writer.py

Lines 415 to 417 in 2b8ba83

	nxentry["entry_identifier"] = self.root["/entry/instrument/bluesky/uid"]
	return nxentry

Handling consists of finding it in the run's metadata, converting it back to a Python dictionary, and parsing the template entries. The parser would need to handling these types of templates:

• links from existing fields or groups to new locations
• creating new groups as directed
• creating constants for attributes or fields

When constructing new groups, the template could specify the NX_class for that group using the NeXus class path notation. For example, monochromator:NXmonochromator specifies an HDF5 group named monochromator with group attribute of NX_class="NXmonochromator".

[prjemian]

The template including the entry_identifier example link above would be:

template = {
    # ...
    "/entry/instrument/bluesky/metadata/run_start_uid": "/entry/entry_identifier",
    # ...
}

[prjemian]

Consider demonstrating the templating system by writing data to the NXmonopd application definition (for raw data) as demonstrated by the the NeXus WONI hypothetical instrument.

[prjemian]

A link from an existing field to a new field in a new group might be:

"/entry/instrument/positioners/m1/value": "/entry/motors:NXnote/motor1"

This would link the existing m1 data to a motor1 field in a new motors group under the /entry group. The new group would be a NXnote group.

• Constant fields could be declared with a = sign:
  • "/entry/constant=": 27.15,
  • "/entry/array=": [1, 2, 3],
  • "/entry/text=": "example",

Attributes (which cannot be used as links) could be declared with a @ sign:

• "/entry/motors/@comment": "one or more motors known this run",

[prjemian]

Refactor the templates structure from a dict to a list. Allow the same "source" to appear more than once. We need that ability to map the same signal source (from a stream) to more than one HDF5 address. Can't do that if the "source" is used as a dictionary key. Here are some examples:

templates = [
    ["/entry/definition=", "NXmonopd"],  # satisfy the NXmonopd definnition
    # /entry/title  already defined
    # /entry/start_time  already defined
    # /entry/instrument/source/type  already defined
    # /entry/instrument/source/name  already defined
    # /entry/instrument/source/probe  already defined
    ["/entry/instrument/crystal:NXcrystal/wavelength=", 1.0],  # simulation
    ["/entry/instrument/bluesky/streams/primary/tth/value", "/entry/instrument/detector/polar_angle"],
    ["/entry/instrument/bluesky/streams/primary/sensor/value", "/entry/instrument/detector/data"],
    ["/entry/instrument/bluesky/metadata/sample_name", "/entry/sample:NXsample/name"],
    ["/entry/sample:NXsample/rotation_angle=", 0],  # sample has not been rotated
    ["/entry/monitor:NXmonitor/mode=", "monitor"],
    ["/entry/monitor:NXmonitor/preset=", 100_000],
    ["/entry/instrument/bluesky/streams/primary/I0/value", "/entry/monitor:NXmonitor/integral"],
    ["/entry/instrument/bluesky/streams/primary/tth/value", "/entry/monopd:NXdata/polar_angle"],
    ["/entry/instrument/bluesky/streams/primary/sensor/value", "/entry/monopd:NXdata/data"],
    ["/entry/monopd/@signal", "data"],
    ["/entry/monopd/@axes", ["polar_angle", ]],
    ["/entry/@default", "monopd"],  # change the default plot group
    ["/entry/monitor/@signal", "integral"],  # in NXmonitor, same as for NXdata groups
]