Improved documentation about time stamped data

manual/source/design.rst

@@ -121,6 +121,9 @@ an array of several dimensions.
 		``data`` (*NX_NUMBER*)
 			Data values from the detector, ``units="NX_ANY"``
 
+In the case of streaming data acquisition, when time stamped values of data are collected, fields can be replaced with :ref:`NXlog` structures of 
+the same name. For example, if time stamped data for wavelength is being streamed, wavelength would not be an array but a :ref:`NXlog` structure.  
+
 
 .. index::
    ! single: field attribute

manual/source/introduction.rst

@@ -14,18 +14,21 @@ to define a common data exchange format for neutron, X-ray, and muon experiments
 NeXus is built on top of the scientific data format HDF5 and adds 
 domain-specific rules for organizing data within HDF5 files in addition 
 to a dictionary of well-defined domain-specific field names. The NeXus 
-data format has two purposes:
+data format has three purposes:
 
 #. *raw data*:
    NeXus defines a format that can 
    serve as a container for all relevant data associated with a scientific 
-   instrument or beamline. 
-   This is a very important use case.
+   instrument or beamline. This is a very important use case. This includes 
+   the case of streaming data acquisition, where time stamped data are logged. 
 #. *processed data*:
+   NeXus also defines standards for processed data. This is data which has underwent some form of data 
+   reduction or data analysis. NeXus allows storing the results of such processing together with 
+   documentation about how the processed data was generated. 
+#. *standards*:
    NeXus defines standards in 
    the form of *application definitions* for the exchange of data 
-   between applications.  NeXus provides structures for raw experimental 
-   data as well as for processed data.  
+   between applications. NeXus provides standards for both raw and processed data.
 
 A community of scientists and computer programmers working in neutron
 and synchrotron facilities around the world came to the conclusion that a 

manual/source/rules.rst

@@ -368,6 +368,22 @@ dimensions ``xraster, yraster, xsize, ysize``.
 .. warning:: Be warned: if you use the 2D rasterisation method with ``xraster, yraster`` you may end up with
              invalid data if the scan is aborted prematurely. This cannot happen if the first method is used.
 
+
+Streaming Data Acquisition And Logging
+======================================
+
+More and more data is collected in streaming mode. This means that time stamped data is logged for one or more inputs, 
+possibly together with detector data. Another use case is the logging of parameters, for example temperature, while a long 
+running data collection is in progress. NeXus covers this case too. There is one simple rule for structuring such files:
+
+Just use the standard NeXus raw data file structure, but replace the corresponding data object 
+with an :ref:`NXlog` or :ref:`NXevent_data` structure of the same name. 
+
+For example, consider your instrument is streaming detector images against a magnetic_field on the sample. In this case both 
+NXsample/magnetic_field and NXdetector/data would become NXlog structures instead of simple arrays i.e. the NXlog structure will 
+have the same name as the NeXus field involved.        
+
+
 NXcollection
 ============
 

manual/source/strategies.rst

@@ -149,10 +149,18 @@ which also allows "energy" to be specified if one is so inclined.
 Strategies: Time-stamped data
 #############################
 *How should I store time-stamped data?*
-Time-stamped data can stored in both :ref:`NXlog` and :ref:`NXevent_data`. 
-:ref:`NXevent_data` is used for storing neutron event data and :ref:`NXlog` would be used 
-for storing any other time-stamped data, e.g. sample temperature, chopper 
-top-dead-centre, motor position etc.
+
+Time-stamped data can be stored in either :ref:`NXlog` and :ref:`NXevent_data` structures. 
+Of the two, :ref:`NXlog` is the most important one, :ref:`NXevent_data` is normally only used for storing detector 
+time of flight event data 
+and :ref:`NXlog` would be used for storing any other time-stamped data, e.g. sample temperature, chopper top-dead-centre, 
+motor position, detector images etc.
+
+Regarding the NeXus file structure to use, there is one simple rule: just use the standard NeXus file structure but insert/replace 
+the fields for streamed data elements through :ref:`NXlog` or :ref:`NXevent_data` structures. For example, consider the 
+collection of detector images against a change in the magnetic field on the sample. Then, both NXsample/magnetic_field and 
+NXdetector/data would be :ref:`NXlog` structures containing the time stamped data. 
+
 
 Both :ref:`NXlog` and :ref:`NXevent_data` have additional support for storing time-stamped 
 data in the form of cues; cues can be used to place markers in the data that allow one to 
@@ -174,7 +182,11 @@ to grab the data between 20 seconds and 40 seconds, and then trim that data to g
 want.
 Obviously in this simple example this does not gain us a lot, but it is easy to see that in a 
 large dataset having appropriately placed cues can save significant computational time when looking 
-up values in a certain time-stamp range.
+up values in a certain time-stamp range. NeXus has actually borrowed the cueing table concept 
+from video file formats where it allows viewing software to quickly access your favourite scene. 
+Correspondingly, cueing in NeXus allows you to quickly access your favourite morsel of time stamped 
+data.  
+
 
 In the NeXus Features repository, the feature `ECB064453EDB096D <https://github.com/nexusformat/features/tree/b0f4862f267844a3f66efa701953e684978b0959/src/recipes/ECB064453EDB096D>`_ 
 shows example code that uses cues to select time-stamped data.