readd rst files

dev_tools/docs/nxdl_index.py

@@ -66,13 +66,22 @@ def nxdl_indices() -> Dict[str, dict]:
         else:
             file = ""
             print("---------++++++++-", section)
+        if file.endswith(("applications/index.rst", "base_classes/index.rst")):
+            rst_lines.append(f"{indentation}em-structure\n")
+            rst_lines.append(f"{indentation}optical-spectroscopy-structure\n")
+            rst_lines.append(f"{indentation}mpes-structure\n")
+            rst_lines.append(f"{indentation}apm-structure\n")
+
         if file.endswith("contributed_definitions/index.rst"):
             rst_lines.append(f"{indentation}em-structure\n")
-            rst_lines.append(f"{indentation}ellipsometry-structure\n")
+            rst_lines.append(f"{indentation}optical-spectroscopy-structure\n")
             rst_lines.append(f"{indentation}mpes-structure\n")
             rst_lines.append(f"{indentation}apm-structure\n")
             rst_lines.append(f"{indentation}transport-structure\n")
+            rst_lines.append(f"{indentation}sts-structure\n")
             rst_lines.append(f"{indentation}cgms-structure\n")
+            rst_lines.append(f"{indentation}icme-structure\n")
+            rst_lines.append(f"{indentation}sample-prep-structure\n")
 
         for cname in sorted(classes):
             rst_lines.append(f"{indentation}{cname}\n")
@@ -110,6 +119,18 @@ def get_nxclass_description(nxdl_file: Path, namespaces) -> str:
 *might* be used in an instance of that class.
 Consider the base classes as a set of *components*
 that are used to construct a data file.
+
+Some contributions are grouped together:
+  :ref:`Optical Spectroscopy <Optical-Spectroscopy-Structure-BC>`
+
+  :ref:`Multi-dimensional Photoemission Spectroscopy <Mpes-Structure-BC>`
+
+  :ref:`Atomprobe Microscopy <Apm-Structure-BC>`
+
+  :ref:`Electron Microscopy <Em-Structure-BC>`
+
+and others are simply listed here:
+
     """,
     # - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     "applications": """
@@ -138,6 +159,19 @@ def get_nxclass_description(nxdl_file: Path, namespaces) -> str:
 In application definitions involving raw data,
 write the raw data in the :ref:`NXinstrument` tree and then link to it
 from the location(s) defined in the relevant application definition.
+
+Some contributions are grouped together:
+  :ref:`Optical Spectroscopy <Optical-Spectroscopy-Structure-APP>`
+
+  :ref:`Multi-dimensional Photoemission Spectroscopy <Mpes-Structure-APP>`
+
+  :ref:`Atomprobe Microscopy <Apm-Structure-APP>`
+
+  :ref:`Electron Microscopy <Em-Structure-APP>`
+
+
+and others are simply listed here:
+
     """,
     # - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     "contributed_definitions": """
@@ -160,7 +194,7 @@ def get_nxclass_description(nxdl_file: Path, namespaces) -> str:
 and acceptance as either a base class or application definition.
 
 Some contributions are grouped together:
-  :ref:`Optical Spectroscopy <Ellipsometry-Structure>`
+  :ref:`Optical Spectroscopy <Optical-Spectroscopy-Structure>`
 
   :ref:`Multi-dimensional Photoemission Spectroscopy <Mpes-Structure>`
 

manual/source/classes/applications/apm-structure.rst

@@ -0,0 +1,138 @@
+.. _Apm-Structure-APP:
+
+=====================
+Atom-probe tomography
+=====================
+
+.. index::
+   IntroductionApm-APP
+   ApmAppDef-APP
+   IntroductionApmParaprobe-APP
+   StatusQuoApm-APP
+   ApmParaprobeAppDef-APP
+   ApmGermanNfdi-APP
+
+
+EXAMPLE FOR DOCUMENTATION OF A GROUP OF APPLICATION DEFINITIONS
+
+
+.. _IntroductionApm-APP:
+
+Introduction
+############
+
+Set of data schemas to describe the acquisition, i.e. measurement side, the extraction of hits from detector raw data,
+steps to compute mass-to-charge state ratios from uncorrected time of flight data, the reconstruction, and the ranging, i.e. identification of peaks in the mass-to-charge-state ratio histogram to detect (molecular) ions.
+The data schemas can be useful to generate data artifacts also for field-ion microscopy experiments.
+
+.. _ApmAppDef-APP:
+
+Application Definition
+######################
+
+    :ref:`NXapm`:
+       A general application definition with many detailed places for leaving metadata and computational steps described which are commonly used when reporting the measurement of atom probe data including also detector hit data, as well as how to proceed with reconstructing atom positions from these data, and how to store details about definitions made, which describe how mass-to-charge-state ratio values are mapped to iontypes in a process called ranging. The structure of the schema has been designed to also document a simulation of an atom probe
+       experiment. Having a combined schema for the measurement and the simulation is beneficial to document that
+       there are many similarities between the measurement and a computer simulation of it.
+
+.. _IntroductionApmParaprobe-APP:
+
+apmtools
+########
+
+The paraprobe-toolbox is an example of an open-source parallelized software for analyzing
+point cloud data, for assessing meshes in 3D continuum space, and for studying the effects of
+parameterization on descriptors of micro- and nanoscale structural features (crystal defects)
+within materials when characterized and studied with atom probe.
+
+The need for a thorough documentation of the tools in not only the paraprobe-toolbox
+was motivated by several needs:
+
+First, users of software would like to better understand and also be able to study for themselves
+which individual parameters and settings for each tool exist and how configuring these
+affects analyses quantitatively. This stresses the aspect how to improve documentation.
+
+Second, scientific software like paraprobe-toolbox implement numerical/algorithmical
+(computational) workflows whereby data coming from multiple input sources
+(like previous analysis results) are processed and carried through more involved analyses
+within several steps inside the tool. The tool then creates output as files. This
+provenance and workflow should be documented.
+
+Individual tools of paraprobe-toolbox are developed in C/C++ and/or Python.
+Provenance tracking is useful as it is one component and requirement for making
+workflows exactly numerically reproducible and thus to enable reproducibility (the "R"
+of the FAIR principles of data stewardship).
+
+For tools of the paraprobe-toolbox each workflow step is a pair or triple of sub-steps:
+1. The creation of a configuration file. 
+2. The actual analysis using the Python/or C/C++ tools. 
+3. The optional analyses/visualization of the results based on data in NeXus/HDF5 files generated by each tool. 
+
+.. _StatusQuoApm-APP:
+
+What has been achieved so far?
+##############################
+
+This proposal summarizes work of members of the FAIRmat project, which is part of the `German
+National Research Data Infrastructure <https://www.nfdi.de/?lang=en>`_. The here detailed
+proposal documents how all tools of the paraprobe-toolbox were modified to generate
+only well-defined configuration files as accepted input and yield specifically formatted output
+files according to the following NeXus application definitions.
+
+Data and metadata between the tools are exchanged with NeXus/HDF5 files. This means that data
+inside HDF5 binary containers are named, formatted, and hierarchically structured according
+to application definitions.
+
+For example the application definition NXapm_paraprobe_config_surfacer specifies
+how a configuration file for the paraprobe-surfacer tool should be formatted
+and which parameters it contains including optionality and cardinality constraints.
+
+Thereby, each config file uses a controlled vocabulary of terms. Furthermore,
+the config files store a SHA256 checksum for each input file. This implements a full
+provenance tracking on the input files along the workflow.
+
+As an example, a user may first range their reconstruction and then compute spatial
+correlation functions. The config file for the ranging tool stores the files
+which hold the reconstructed ion position and ranging definitions.
+The ranging tool generates a results file with the labels of each molecular ion.
+This results file is formatted according to the tool-specific `results`
+application definition. The generated results file and the reconstruction is
+imported by the spatial statistics tool which again keeps track of all files
+and reports its results in a spatial statistics tool results file.
+
+This design makes it possible to rigorously trace which numerical results were achieved
+with specific inputs and settings using specifically-versioned tools. Noteworthy
+this includes Y-junction on a graph which is where multiple input sources are
+combined to generate new results.
+
+We are convinced that defining, documenting, using, and sharing application definitions
+is useful and future-proof strategy for software development and data analyses as it enables
+automated provenance tracking which happens silently in the background.
+
+Base classes have been defined to group common pieces of information for each tool of the
+toolbox. For each tool we define a pair of base classes. One for the configuration (input) side
+and one for the results (output) side:
+
+
+.. _ApmParaprobeAppDef-APP:
+
+Application Definitions
+#######################
+
+NXapm_paraprobe application definitions are in fact pairs of application definitions.
+One for the configuration (input) side and one for the results (output) side. For each
+tool one such pair is proposed:
+
+
+.. _ApmGermanNfdi-APP:
+
+Joint work German NFDI consortia NFDI-MatWerk and FAIRmat
+#######################################################################
+
+Members of the NFDI-MatWerk and the FAIRmat consortium of the German National Research Data Infrastructure
+are working together within the Infrastructure Use Case IUC09 of the NFDI-MatWerk project to work on examples
+how software tools in both consortia become better documented and interoperable to use. Within this project,
+we have also added the `CompositionSpace tool that has been developed at the Max-Planck-Institut für Eisenforschung
+GmbH in Düsseldorf <https://github.com/eisenforschung/CompositionSpace>`_ by A. Saxena et al.
+
+Specifically, within the IUC09 we refactored the code base behind the publication `A. Saxena et al. <https://dx.doi.org/10.1093/micmic/ozad086>`_ to better document its configuration, here as an example implemented like for  the above-mentioned paraprobe-toolbox using NeXus:

manual/source/classes/applications/em-structure.rst

@@ -0,0 +1,42 @@
+.. _Em-Structure-APP:
+
+===================
+Electron microscopy
+===================
+
+.. index::
+   IntroductionEm-APP
+   EmAppDef-APP
+
+
+EXAMPLE FOR DOCUMENTATION OF A GROUP OF APPLICATION DEFINITIONS
+
+
+.. _IntroductionEm-APP:
+
+Introduction
+############
+
+A set of data schemas is proposed to describe components of an electron microscope and its eventually available focused-ion beam functionalities.
+The data schemas were designed from the perspective of how electron microscopes are used by colleagues in the materials-science-branch of electron microscopy.
+We realize that the biology-/bio-materials/omics-branch of electron microscopy is eventually in an already more mature state of discussion with respect
+to data management practices. In what follows, the focus is on the usage of electron microscopy in condensed-matter physics, chemical physics of solids,
+and materials engineering applications. As many of the components of electron microscopes used in the bio-materials communities are the same or at least many
+components are very similar, it is likely that the here presented schema definitions can also inspire discussion and exchange with the bio-materials community.
+Partner consortia in the German National Research Data Infrastructure are here e.g. NFDI-BioImage, NFDI-Microbiota, NFDI4Health, and e.g. NFDI-Neuro.
+
+Electron microscopes are functionally very customizable tools: Examples include multi-signal/-modal analyses which are frequently realized as on-the-fly computational analyses, regularly switching between GUI-based instrument control, computational steps, and more and more using high-throughput stream-based processing. Also artificial intelligence methods are increasingly used and are becoming more closely interconnected with classical modes of controlling the instrument and perform data processing. A challenge in electron microscopy is that these steps are often executed within commercial integrated control and analysis software. This makes it difficult to keep track of workflows in a technology-partner-agnostic, i.e. interdisciplinary manner.
+
+.. _EmAppDef-APP:
+
+Application Definitions
+#######################
+
+We acknowledge that it can be difficult to agree on a single application definition which is generally enough applicable yet not unnecessarily complex and useful for applications across a variety of instruments, technology partners, and instrument use cases. In what follows, the proposal conceptualizes first the basic components of an electron microscope and the usual workflow of how an electron microscope is used for collecting data with detectors via probing radiation-specimen-matter interaction mechanisms.
+
+In summary, scientists place a specimen/sample into the microscope, calibrate the instrument, take measurements, and may perform experiments, prepare their specimens with a focused ion beam, calibrate again, and take other measurements, before their session on the instrument ends. In between virtually all of these steps data are collected and stream in from different detectors probing different physical mechanisms of the interaction between electrons or other types of radiation with the specimen.
+
+A microscope session ends with the scientist removing the specimen from the instrument or parking it so that the next user can start a session. Occasionally, service technicians perform calibrations and maintenance which also can be described as a session on the microscope. We have provided base classes to describe these steps and events and an application definition for electron microscopy:
+
+    :ref:`NXem`:
+        An application definition which explores the possibilities of electron microscopes.

manual/source/classes/applications/mpes-structure.rst

@@ -0,0 +1,33 @@
+.. _Mpes-Structure-APP:
+
+=======================================
+Photoemission & core-level spectroscopy
+=======================================
+
+.. index::
+   IntroductionMpes-APP
+   MpesAppDef-APP
+
+
+EXAMPLE FOR DOCUMENTATION OF A GROUP OF APPLICATION DEFINITIONS
+
+
+. _IntroductionMpes-APP:
+
+Introduction
+############
+
+Set of data storage objects to describe multidimensional photoemission (MPES) experiments including x-ray photoelectron spectroscopy (XPS), ultraviolet photoelectron spectroscopy (UPS),
+hard x-ray photoelectron spectroscopy (HAXPES), angle-resolved photoemission spectroscopy (ARPES), two-photon photoemission (2PPE) 
+and photoemission electron microscopy (PEEM). Also includes descriptors for advanced specializations, such as spin-resolution, time resolution, 
+near-ambient pressure conditions, dichroism etc.
+
+.. _MpesAppDef-APP:
+
+Application Definitions
+#######################
+
+:ref:`NXmpes`:
+   A general application definition with minimalistic metadata requirements, apt to describe all photoemission experiments.
+
+

manual/source/classes/applications/optical-spectroscopy-structure.rst

@@ -0,0 +1,66 @@
+.. _Optical-Spectroscopy-Structure-APP:
+
+====================
+Optical Spectroscopy
+====================
+
+.. index::
+   Ellipsometry-APP
+   Raman-APP
+   DispersiveMaterial-APP
+
+
+EXAMPLE FOR DOCUMENTATION OF A GROUP OF APPLICATION DEFINITIONS
+
+
+.. _Ellipsometry-APP:
+
+Ellipsometry
+############
+
+Ellipsometry is an optical characterization method to describe optical properties of interfaces and thickness of films.
+The measurements are based on determining how the polarization state of light changes upon transmission and reflection.
+Interpretation is based on Fresnel equations and numerical models of the optical properties of the materials.
+
+In the application definition, we provide a minimum set of description elements allowing for a reproducible recording of ellipsometry measurements. 
+
+.. _Raman-APP:
+
+Raman
+############
+
+Raman spectroscopy is a characterization method to analyze vibrational properties for liquids, gases, or solids. 
+The measurements is based on the inelastic light scattering due to the material's vibrations.
+Interpretation can be done based on peaks, which represent the phonon properties (intensity, center, width).
+
+The application definition contains a minimum of descriptive elements required to understand Raman spectroscopy measurements.
+
+
+Application Definitions
+-----------------------
+
+    :ref:`NXoptical_spectroscopy`:
+       A generic application definition for spectroscopy measurements. This includes in particular ellipsometry and Raman spectroscopy measurements, but also other techniques such as photoluminescence, transmission, and reflection measurements. The requirements are: (i) an incident photon beam, (ii) a detector to measure scattered/emitted photons, and (iii) a sample.
+
+    :ref:`NXellipsometry`:
+       An application definition for ellipsometry measurements, including complex systems up to variable angle spectroscopic ellipsometry.
+
+    :ref:`NXraman`:
+       An application definition for Raman spectroscopy measurements.
+
+.. _DispersiveMaterial-APP:
+
+Dispersive Material
+###################
+
+A dispersive material is a description for the optical dispersion of materials.
+This description may be used to store optical model data from an ellipsometric analysis 
+(or any other technique) or to build a database of optical constants for optical properties of materials.
+
+Application Definition
+----------------------
+
+    :ref:`NXdispersive_material`:
+       An application definition to describe the dispersive properties of a material.
+       The material may be isotropic, uniaxial or biaxial. Hence, it may contain up
+       to three dispersive functions or tables.