From c18cf0c720beb37471f5c23bf18a11590f5906f6 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Sat, 31 Dec 2022 20:58:50 +0100 Subject: [PATCH 01/30] Intermediate state adding ranged properties. --- optimade.rst | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/optimade.rst b/optimade.rst index 4131c8bf2..a6ce142b8 100644 --- a/optimade.rst +++ b/optimade.rst @@ -21,6 +21,7 @@ OPTIMADE API specification v1.1.0~develop entry : names of type of resources, served via OPTIMADE, pertaining to data in a database. property : data item that belongs to an entry. + ranged_property : A property that can be returned in pieces and that supports slicing. val : value examples that properties can be. :val: is ONLY used when referencing values of actual properties, i.e., information that belongs to the database. type : data type of values. @@ -67,6 +68,8 @@ OPTIMADE API specification v1.1.0~develop .. role:: property(literal) +.. role:: ranged_property(literal) + .. role:: val(literal) .. role:: type(literal) @@ -420,6 +423,109 @@ For example, the following query can be sent to API implementations `exmpl1` and :filter:`filter=_exmpl1_band_gap<2.0 OR _exmpl2_band_gap<2.5` + +Ranged_Properties +----------------- + +- **Description**: In some cases the value of a property can be so large that it becomes unwieldy to process them. + In this case defining a property as a ranged_property can be a good solution. + Ranged properties support slicing, so the client can request that only some of the values need to be returned. Likewise, the server can use paging to return the property in multiple parts. +How to retrieve the data of ranged properties + +- **Type**: dictionary with keys: + + - :property:`serialization_format`: string (REQUIRED) + - :property:`values`: List of any data type (REQUIRED) + - :property:`nvalues`: integer (REQUIRED) + - :property:`range_id`: string(OPTIONAL) + - :property:`n_range_dim`: integer(REQUIRED) + - :property:`dim_range`: list of integers (REQUIRED) + - :property:`offset_linear`: list of floats (OPTIONAL) + - :property:`step_size_linear`: list of floats (OPTIONAL) + - :property:`offset_regular`: integer (OPTIONAL) + - :property:`step_size_regular`: list of integers (OPTIONAL) + - :property:`indexes`: list of integers(OPTIONAL) + - :property:`average`: float (OPTIONAL) + - :property:`min`: list of float or integer (OPTIONAL) + - :property:`max`: list of float or integer (OPTIONAL) + - :property:`set`: List of Any data type (OPTIONAL) + +- **Requirements/Conventions**: + - **Support**: OPTIONAL support in implementations. + - Ranged_properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first. + - **examples**: :ranged_property:`_ranged_cartesian_sites_positions`, :ranged_property:`_exmpl_ranged_field` + - If the part of the property name after the `_ranged_` prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. + - **Query**: Queries on the dictionary fields SHOULD be supported, with the exception of the values field for which Querying is OPTIONAL. + - **Keys**: + + - **serialization_format**: To improve the compactness of the data there are several ways to show to which index a value belongs. + This is specified by the :property:`serialization_format`. + + - **Type**: string + - **Requirements/Conventions**: This field MUST be present. + - **Possible values**: + + - **linear**: The value is a linear function of the index(es). + This function is defined by :property:`offset_linear` and :property:`step_size_linear`. + - **regular**: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` as the first frame. + - **custom**: A separate list with range indexes is defined in the field :property:`indexes` to indicate to which index a value belongs. + + - **values**: The values belonging to this property. + The format of this field depends on the property. + + - **Type**: List of Any + - **Requirements/Conventions**: The :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. + + - **nvalues**: The number of values in the field :property:`values`. + + - **Type**: integer + - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. + + - **range_id**: An identifier that uniquely identifies a range. It can be used to show that a property uses the same range as another property. + For example, when data of a MD trajectory is shared it could be used to indicate that the energies and the cartesian_site_positions use the same ranges and thereby which energy belongs to which set of cartesian_site_positions. + + - **Type**: string + - **Requirements/Conventions**: OPTIONAL. + + - **dim_range**: The dimensions of the range in each dimension. It MUST have the same value for properties that have the same :property:`range_id`. + - **Type**: list of integers + - **Requirements/Conventions**: REQUIRED; + + - **offset_linear**: If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. + + - **Type**: float + - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value MUST NOT be present. + The default value is 0.0. + + - **step_size_linear**:If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. + e.g. If at frame 3 the value of the property is 0.6 and :property:`step_size_linear` = [0.2] than at frame 4 the value of the property will be 0.8. + + - **Type**: list of float + - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "linear". + Otherwise it MUST NOT be present. + + - **offset_regular**: If :property:`serialization_format` is set to :val:` "regular"` this property gives the indexes of the first value. + + - **Type**: list of integers + - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value MUST NOT be present. + The default value is 1 in every dimension. + + - **step_size_regular**: If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension. + + - **Type**: list of integers + - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. + Otherwise it MUST NOT be present. + + - **indexes**: If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. + - **Type**: List of list of integers + - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "custom". + Otherwise it MUST NOT be present. The order of the values must be the same as those in :property:`values`. + + - **Examples**: + + - :val:`[[0,20,78],[3,22,67]]` + + Responses ========= From 6afcf9c1c1e32180e7769253a7f5b954c23b6c98 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 6 Jan 2023 18:55:31 +0100 Subject: [PATCH 02/30] first draft for ranged properties. --- optimade.rst | 184 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 125 insertions(+), 59 deletions(-) diff --git a/optimade.rst b/optimade.rst index 8a051743e..e230a5971 100644 --- a/optimade.rst +++ b/optimade.rst @@ -68,7 +68,7 @@ OPTIMADE API specification v1.2.0-rc.1 .. role:: property(literal) -.. role:: ranged_property(literal) +.. role:: ranged-property(literal) .. role:: val(literal) @@ -449,22 +449,24 @@ For example, the following query can be sent to API implementations `exmpl1` and Ranged_Properties ----------------- -- **Description**: In some cases the value of a property can be so large that it becomes unwieldy to process them. - In this case defining a property as a ranged_property can be a good solution. +- **Description**: Ranged properties are used when the value of a property is so large that it can become unwieldy to process. + They also provide a method to correlate the values of two separate properties. + The meta data is returned by default, the values are only returned when specifically requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. Ranged properties support slicing, so the client can request that only some of the values need to be returned. Likewise, the server can use paging to return the property in multiple parts. -How to retrieve the data of ranged properties + In that case a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. + - **Type**: dictionary with keys: - :property:`serialization_format`: string (REQUIRED) - :property:`values`: List of any data type (REQUIRED) - :property:`nvalues`: integer (REQUIRED) - - :property:`range_id`: string(OPTIONAL) + - :property:`range_ids`: list of strings (OPTIONAL) - :property:`n_range_dim`: integer(REQUIRED) - :property:`dim_range`: list of integers (REQUIRED) - :property:`offset_linear`: list of floats (OPTIONAL) - :property:`step_size_linear`: list of floats (OPTIONAL) - - :property:`offset_regular`: integer (OPTIONAL) + - :property:`offset_regular`: list of integers (OPTIONAL) - :property:`step_size_regular`: list of integers (OPTIONAL) - :property:`indexes`: list of integers(OPTIONAL) - :property:`average`: float (OPTIONAL) @@ -473,79 +475,131 @@ How to retrieve the data of ranged properties - :property:`set`: List of Any data type (OPTIONAL) - **Requirements/Conventions**: - - **Support**: OPTIONAL support in implementations. - - Ranged_properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first. - - **examples**: :ranged_property:`_ranged_cartesian_sites_positions`, :ranged_property:`_exmpl_ranged_field` - - If the part of the property name after the `_ranged_` prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. - - **Query**: Queries on the dictionary fields SHOULD be supported, with the exception of the values field for which Querying is OPTIONAL. - - **Keys**: - - **serialization_format**: To improve the compactness of the data there are several ways to show to which index a value belongs. - This is specified by the :property:`serialization_format`. + - **Support**: OPTIONAL support in implementations. + - Ranged_properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first. + - If the part of the property name after the `_ranged_` prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. + - By default only the metadata SHOULD be returned (i.e. all the fields except :property:`values`). + The :property:`values` SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. + - The data from the :property:`values` is only returned when requested via the :query-param: `property_ranges`, which is described in the section + + - **examples**: :ranged-property:`_ranged_cartesian_sites_positions`, :ranged-property:`_exmpl_ranged_field` + - **Query**: Queries on the dictionary fields SHOULD be supported, with the exception of the values field for which querying is OPTIONAL. + - **Keys**: + + - **serialization_format**: To improve the compactness of the data there are several ways to show to which index a value belongs. + This is specified by the :property:`serialization_format`. + + - **Type**: string + + - **Requirements/Conventions**: This field MUST be present. + - **Possible values**: + + - **linear**: The value is a linear function of the index(es). + This function is defined by :property:`offset_linear` and :property:`step_size_linear`. + - **regular**: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` as the first frame. + - **custom**: A separate list with range indexes is defined in the field :property:`indexes` to indicate to which index a value belongs. + + - **values**: The values belonging to this property. + The format of this field depends on the property. + + - **Type**: List of Any + - **Requirements/Conventions**: The :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. - - **Type**: string - - **Requirements/Conventions**: This field MUST be present. - - **Possible values**: + - **nvalues**: The number of values in the field :property:`values`. - - **linear**: The value is a linear function of the index(es). - This function is defined by :property:`offset_linear` and :property:`step_size_linear`. - - **regular**: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` as the first frame. - - **custom**: A separate list with range indexes is defined in the field :property:`indexes` to indicate to which index a value belongs. + - **Type**: integer + - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. - - **values**: The values belonging to this property. - The format of this field depends on the property. + - **range_ids**: A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range. + For example, when data of a MD trajectory is shared it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. i.e. which energy belongs to which set of cartesian_site_positions. - - **Type**: List of Any - - **Requirements/Conventions**: The :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. + - **Type**: list of string + - **Requirements/Conventions**: OPTIONAL. - - **nvalues**: The number of values in the field :property:`values`. + - **dim_range**: The dimensions of the range in each dimension. + - **Type**: list of integers + - **Requirements/Conventions**: REQUIRED; - - **Type**: integer - - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. + - **offset_linear**: If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. - - **range_id**: An identifier that uniquely identifies a range. It can be used to show that a property uses the same range as another property. - For example, when data of a MD trajectory is shared it could be used to indicate that the energies and the cartesian_site_positions use the same ranges and thereby which energy belongs to which set of cartesian_site_positions. + - **Type**: float + - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value MUST NOT be present. + The default value is 0 in each dimension. - - **Type**: string - - **Requirements/Conventions**: OPTIONAL. + - **step_size_linear**: If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. + e.g. If at index[1,1] the value is 0 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be . - - **dim_range**: The dimensions of the range in each dimension. It MUST have the same value for properties that have the same :property:`range_id`. - - **Type**: list of integers - - **Requirements/Conventions**: REQUIRED; + - **Type**: list of float + - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "linear". + Otherwise it MUST NOT be present. - - **offset_linear**: If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. + - **offset_regular**: If :property:`serialization_format` is set to :val:` "regular"` this property gives the indexes of the first value. - - **Type**: float - - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value MUST NOT be present. - The default value is 0.0. + - **Type**: list of integers + - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value MUST NOT be present. + The default value is 1 in every dimension. - - **step_size_linear**:If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. - e.g. If at frame 3 the value of the property is 0.6 and :property:`step_size_linear` = [0.2] than at frame 4 the value of the property will be 0.8. + - **step_size_regular**: If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension. - - **Type**: list of float - - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "linear". - Otherwise it MUST NOT be present. + - **Type**: list of integers + - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. + Otherwise it MUST NOT be present. - - **offset_regular**: If :property:`serialization_format` is set to :val:` "regular"` this property gives the indexes of the first value. + - **indexes**: If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. - - **Type**: list of integers - - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value MUST NOT be present. - The default value is 1 in every dimension. + - **Type**: List of lists of integers + - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "custom". + Otherwise it MUST NOT be present. The order of the values must be the same as those in :property:`values`. - - **step_size_regular**: If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension. + - **Example**: - - **Type**: list of integers - - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. - Otherwise it MUST NOT be present. + .. code:: jsonc - - **indexes**: If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. - - **Type**: List of list of integers - - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "custom". - Otherwise it MUST NOT be present. The order of the values must be the same as those in :property:`values`. + { + "_ranged_cartesian_site_positions": { + "n_range_dim": 3, + "dim_range": [100,3,3], + "range_ids": ["mdsteps","particles","xyz"], + "serialization_format": "regular", + "offset_regular": [0,0,0], + "step_size_regular": [1,1,1], + "nvalues": 900, + "values": [[[2.36,5.36,9.56],[7.24,3.58,0.56],[8.12,6.95,4.56]], + [[2.38,5.37,9.56],[7.24,3.57,0.58],[8.11,6.93,4.58]], + [[2.39,5.38,9.55],[7.23,3.57,0.59],[8.10,6.93,4.57]], + // ... + ], + }, + "_ranged_species_at_sites": { + "n_range_dim": 1, + "dim_range": [3], + "range_ids": ["particles"], + "serialization_format": "regular", + "offset_regular": [0], + "step_size_regular": [1], + "nvalues": 3, + "values": ["He", "Ne","Ar"], + }, + "_exmpl_ranged_time":{ + "n_range_dim": 1, + "dim_range": [100], + "range_ids": ["mdsteps"], + "serialization_format": "linear", + "step_size_linear": 0.2, + }, + "_exmpl_ranged_thermostat": { + "n_range_dim": 1, + "dim_range": [100], + "range_ids": ["mdsteps"], + "serialization_format": "custom", + "nvalues": 3, + "values": [20,40,60] + "indexes": [0,20,80] + } - - **Examples**: + } - - :val:`[[0,20,78],[3,22,67]]` Responses @@ -960,7 +1014,6 @@ Standard OPTIONAL URL query parameters standardized by the JSON API specificatio The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON API 1.0 specification. The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. An example is shown in section `Entry Listing Info Endpoints`_. - - **include**: A server MAY implement the JSON API concept of returning `compound documents `__ by utilizing the :query-param:`include` query parameter as specified by `JSON API 1.0 `__. All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. @@ -986,6 +1039,19 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` +- **property_ranges**: Specifies which ranges should be returned for ranged properties. + It consists of a property name directly followed by the range that should be returned. + Ranges can be specified for multiple properties by separating them with a comma. + The ranges are 1 based. i.e. The first value is value 1. + The first value of the range specifies the first index in that dimension for which values should be returned. + The second value specifies the last index for which values should be be returned. + The third value specifies the step size. + Databases that use ranged properties SHOULD support this query parameter. + Example: + + If there would be a structure with id: id_ property :ranged-property:`range_test` with the values :val:`[[9.64, 7.52, 0.69, 5.69],[4.82, 8.35, 3.26, 3.25],[4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=range_test[[1, 3, 2],[2, 3, 1]]` + will return the value: :val:`[[7.52, 0.69],[2.78, 7.87]]` + Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". From b597a67ddb5a3fc79c3d900e049435fad6dc35d1 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 9 Jan 2023 16:38:50 +0100 Subject: [PATCH 03/30] Removed average, set, min and max fields for now as these become quite complicated when multiple dimensions are involved. --- optimade.rst | 50 ++++++++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/optimade.rst b/optimade.rst index e83cce590..d31b79de5 100644 --- a/optimade.rst +++ b/optimade.rst @@ -446,33 +446,29 @@ For example, the following query can be sent to API implementations `exmpl1` and :filter:`filter=_exmpl1_band_gap<2.0 OR _exmpl2_band_gap<2.5` -Ranged_Properties +Ranged Properties ----------------- -- **Description**: Ranged properties are used when the value of a property is so large that it can become unwieldy to process. - They also provide a method to correlate the values of two separate properties. - The meta data is returned by default, the values are only returned when specifically requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. - Ranged properties support slicing, so the client can request that only some of the values need to be returned. Likewise, the server can use paging to return the property in multiple parts. +- **Description**: Ranged properties support slicing, so the client can request that only some of the values need to be returned. + Likewise, the server can use paging to return the property in multiple parts. + This can be useful for properties that are so large that it can be inconvenient to return them in a single response. In that case a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. - + They also provide a method to correlate the values of two ranged properties. + The meta data is returned by default, the values are only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. - **Type**: dictionary with keys: - :property:`serialization_format`: string (REQUIRED) - - :property:`values`: List of any data type (REQUIRED) - - :property:`nvalues`: integer (REQUIRED) + - :property:`values`: List of any data type (OPTIONAL) + - :property:`nvalues`: integer (OPTIONAL) - :property:`range_ids`: list of strings (OPTIONAL) - - :property:`n_range_dim`: integer(REQUIRED) - - :property:`dim_range`: list of integers (REQUIRED) + - :property:`n_dim`: integer(REQUIRED) + - :property:`dim_size`: list of integers (REQUIRED) - :property:`offset_linear`: list of floats (OPTIONAL) - :property:`step_size_linear`: list of floats (OPTIONAL) - :property:`offset_regular`: list of integers (OPTIONAL) - :property:`step_size_regular`: list of integers (OPTIONAL) - - :property:`indexes`: list of integers(OPTIONAL) - - :property:`average`: float (OPTIONAL) - - :property:`min`: list of float or integer (OPTIONAL) - - :property:`max`: list of float or integer (OPTIONAL) - - :property:`set`: List of Any data type (OPTIONAL) + - :property:`indexes`: list of list of integers(OPTIONAL) - **Requirements/Conventions**: @@ -481,9 +477,7 @@ Ranged_Properties - If the part of the property name after the `_ranged_` prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. - By default only the metadata SHOULD be returned (i.e. all the fields except :property:`values`). The :property:`values` SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. - - The data from the :property:`values` is only returned when requested via the :query-param: `property_ranges`, which is described in the section - - **examples**: :ranged-property:`_ranged_cartesian_sites_positions`, :ranged-property:`_exmpl_ranged_field` - **Query**: Queries on the dictionary fields SHOULD be supported, with the exception of the values field for which querying is OPTIONAL. - **Keys**: @@ -517,7 +511,7 @@ Ranged_Properties - **Type**: list of string - **Requirements/Conventions**: OPTIONAL. - - **dim_range**: The dimensions of the range in each dimension. + - **dim_size**: The dimensions of the range in each dimension. - **Type**: list of integers - **Requirements/Conventions**: REQUIRED; @@ -552,14 +546,14 @@ Ranged_Properties - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "custom". Otherwise it MUST NOT be present. The order of the values must be the same as those in :property:`values`. - - **Example**: +- **Example**: .. code:: jsonc { "_ranged_cartesian_site_positions": { - "n_range_dim": 3, - "dim_range": [100,3,3], + "n_dim": 3, + "dim_size": [100,3,3], "range_ids": ["mdsteps","particles","xyz"], "serialization_format": "regular", "offset_regular": [0,0,0], @@ -572,8 +566,8 @@ Ranged_Properties ], }, "_ranged_species_at_sites": { - "n_range_dim": 1, - "dim_range": [3], + "n_dim": 1, + "dim_size": [3], "range_ids": ["particles"], "serialization_format": "regular", "offset_regular": [0], @@ -582,20 +576,20 @@ Ranged_Properties "values": ["He", "Ne","Ar"], }, "_exmpl_ranged_time":{ - "n_range_dim": 1, - "dim_range": [100], + "n_dim": 1, + "dim_size": [100], "range_ids": ["mdsteps"], "serialization_format": "linear", "step_size_linear": 0.2, }, "_exmpl_ranged_thermostat": { - "n_range_dim": 1, - "dim_range": [100], + "n_dim": 1, + "dim_size": [100], "range_ids": ["mdsteps"], "serialization_format": "custom", "nvalues": 3, "values": [20,40,60] - "indexes": [0,20,80] + "indexes": [[0],[20],[80]] } } From 37db8781832330eb9c553b3c0d79675ddc4f98c9 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 9 Jan 2023 18:26:14 +0100 Subject: [PATCH 04/30] Small corrections. --- optimade.rst | 68 ++++++++++++++++++++++++++++------------------------ 1 file changed, 37 insertions(+), 31 deletions(-) diff --git a/optimade.rst b/optimade.rst index d31b79de5..139f680f9 100644 --- a/optimade.rst +++ b/optimade.rst @@ -464,7 +464,7 @@ Ranged Properties - :property:`range_ids`: list of strings (OPTIONAL) - :property:`n_dim`: integer(REQUIRED) - :property:`dim_size`: list of integers (REQUIRED) - - :property:`offset_linear`: list of floats (OPTIONAL) + - :property:`offset_linear`: float (OPTIONAL) - :property:`step_size_linear`: list of floats (OPTIONAL) - :property:`offset_regular`: list of integers (OPTIONAL) - :property:`step_size_regular`: list of integers (OPTIONAL) @@ -473,10 +473,10 @@ Ranged Properties - **Requirements/Conventions**: - **Support**: OPTIONAL support in implementations. - - Ranged_properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first. + - Ranged properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first. - If the part of the property name after the `_ranged_` prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. - By default only the metadata SHOULD be returned (i.e. all the fields except :property:`values`). - The :property:`values` SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. + The :property:`values` field SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. - **Query**: Queries on the dictionary fields SHOULD be supported, with the exception of the values field for which querying is OPTIONAL. - **Keys**: @@ -489,16 +489,16 @@ Ranged Properties - **Requirements/Conventions**: This field MUST be present. - **Possible values**: - - **linear**: The value is a linear function of the index(es). + - **linear**: The value is a linear function of the indexes. This function is defined by :property:`offset_linear` and :property:`step_size_linear`. - - **regular**: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` as the first frame. - - **custom**: A separate list with range indexes is defined in the field :property:`indexes` to indicate to which index a value belongs. + - **regular**: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` indicating the index of the first value. + - **custom**: A separate list with indexes is defined in the field :property:`indexes` to indicate to which index each value belongs. - **values**: The values belonging to this property. - The format of this field depends on the property. + The format of this field depends on the property for which data is stored. - **Type**: List of Any - - **Requirements/Conventions**: The :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. + - **Requirements/Conventions**: The property :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. - **nvalues**: The number of values in the field :property:`values`. @@ -508,43 +508,49 @@ Ranged Properties - **range_ids**: A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range. For example, when data of a MD trajectory is shared it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. i.e. which energy belongs to which set of cartesian_site_positions. - - **Type**: list of string - - **Requirements/Conventions**: OPTIONAL. + - **Type**: list of strings + - **Requirements/Conventions**: OPTIONAL + + - **n_dim**: The number of dimensions this property has. + - **Type**: integer + - **Requirements/Conventions**: REQUIRED + It MUST be equal to the length of the dim_size field. - **dim_size**: The dimensions of the range in each dimension. + - **Type**: list of integers - - **Requirements/Conventions**: REQUIRED; + - **Requirements/Conventions**: REQUIRED - **offset_linear**: If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. - **Type**: float - - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value MUST NOT be present. + - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value SHOULD NOT be present. The default value is 0 in each dimension. - **step_size_linear**: If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. - e.g. If at index[1,1] the value is 0 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be . + e.g. If the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8 . - **Type**: list of float - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "linear". - Otherwise it MUST NOT be present. + Otherwise it SHOULD NOT be present. - - **offset_regular**: If :property:`serialization_format` is set to :val:` "regular"` this property gives the indexes of the first value. + - **offset_regular**: If :property:`serialization_format` is set to :val:`"regular"` this property gives the indexes of the first value. - **Type**: list of integers - - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value MUST NOT be present. + - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value SHOULD NOT be present. The default value is 1 in every dimension. - **step_size_regular**: If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension. - **Type**: list of integers - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. - Otherwise it MUST NOT be present. + Otherwise it SHOULD NOT be present. - **indexes**: If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. - **Type**: List of lists of integers - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "custom". - Otherwise it MUST NOT be present. The order of the values must be the same as those in :property:`values`. + Otherwise it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. - **Example**: @@ -553,15 +559,15 @@ Ranged Properties { "_ranged_cartesian_site_positions": { "n_dim": 3, - "dim_size": [100,3,3], + "dim_size": [100, 3, 3], "range_ids": ["mdsteps","particles","xyz"], "serialization_format": "regular", - "offset_regular": [0,0,0], - "step_size_regular": [1,1,1], + "offset_regular": [1, 1, 1], + "step_size_regular": [1, 1, 1], "nvalues": 900, - "values": [[[2.36,5.36,9.56],[7.24,3.58,0.56],[8.12,6.95,4.56]], - [[2.38,5.37,9.56],[7.24,3.57,0.58],[8.11,6.93,4.58]], - [[2.39,5.38,9.55],[7.23,3.57,0.59],[8.10,6.93,4.57]], + "values": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], + [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], + [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]], // ... ], }, @@ -573,7 +579,7 @@ Ranged Properties "offset_regular": [0], "step_size_regular": [1], "nvalues": 3, - "values": ["He", "Ne","Ar"], + "values": ["He", "Ne", "Ar"], }, "_exmpl_ranged_time":{ "n_dim": 1, @@ -588,10 +594,9 @@ Ranged Properties "range_ids": ["mdsteps"], "serialization_format": "custom", "nvalues": 3, - "values": [20,40,60] - "indexes": [[0],[20],[80]] + "values": [20, 40, 60] + "indexes": [[0], [20], [80]] } - } @@ -1033,18 +1038,19 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` -- **property_ranges**: Specifies which ranges should be returned for ranged properties. +- **property\_ranges**: Specifies which ranges should be returned for ranged properties. It consists of a property name directly followed by the range that should be returned. Ranges can be specified for multiple properties by separating them with a comma. The ranges are 1 based. i.e. The first value is value 1. The first value of the range specifies the first index in that dimension for which values should be returned. The second value specifies the last index for which values should be be returned. - The third value specifies the step size. + The third value specifies the step size. Databases that use ranged properties SHOULD support this query parameter. Example: - If there would be a structure with id: id_ property :ranged-property:`range_test` with the values :val:`[[9.64, 7.52, 0.69, 5.69],[4.82, 8.35, 3.26, 3.25],[4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=range_test[[1, 3, 2],[2, 3, 1]]` + If there would be a structure with id: id_ property :ranged-property:`_ranged_test_field` with the values :val:`[[9.64, 7.52, 0.69, 5.69],[4.82, 8.35, 3.26, 3.25],[4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2],[2, 3, 1]]` will return the value: :val:`[[7.52, 0.69],[2.78, 7.87]]` + Multiple ranges can be requested in one query. e.g. :query-paramproperty_ranges=_ranged_test_field[[1, 3, 2],[2, 3, 1]], _ranged_other_field[]` Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". From a832751828db27850c5abedced2f7f69a0f025fd Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 11 Jan 2023 16:42:31 +0100 Subject: [PATCH 05/30] Added how to treat missing values for requested range. --- optimade.rst | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/optimade.rst b/optimade.rst index 139f680f9..953b44276 100644 --- a/optimade.rst +++ b/optimade.rst @@ -454,7 +454,7 @@ Ranged Properties This can be useful for properties that are so large that it can be inconvenient to return them in a single response. In that case a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. They also provide a method to correlate the values of two ranged properties. - The meta data is returned by default, the values are only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. + The metadata is returned by default, the values are only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. - **Type**: dictionary with keys: @@ -462,26 +462,26 @@ Ranged Properties - :property:`values`: List of any data type (OPTIONAL) - :property:`nvalues`: integer (OPTIONAL) - :property:`range_ids`: list of strings (OPTIONAL) - - :property:`n_dim`: integer(REQUIRED) + - :property:`n_dim`: integer (REQUIRED) - :property:`dim_size`: list of integers (REQUIRED) - :property:`offset_linear`: float (OPTIONAL) - :property:`step_size_linear`: list of floats (OPTIONAL) - :property:`offset_regular`: list of integers (OPTIONAL) - :property:`step_size_regular`: list of integers (OPTIONAL) - - :property:`indexes`: list of list of integers(OPTIONAL) + - :property:`indexes`: list of lists of integers (OPTIONAL) - **Requirements/Conventions**: - **Support**: OPTIONAL support in implementations. - Ranged properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first. - If the part of the property name after the `_ranged_` prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. - - By default only the metadata SHOULD be returned (i.e. all the fields except :property:`values`). - The :property:`values` field SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. + - By default, only the metadata SHOULD be returned (i.e. all the fields except :property:`values` and :property:`indexes`). + The :property:`values` and :property:`indexes` fields SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. - - **Query**: Queries on the dictionary fields SHOULD be supported, with the exception of the values field for which querying is OPTIONAL. + - **Query**: Queries on the dictionary fields SHOULD be supported, except for the :property:`values` and :property:`indexes` fields for which querying is OPTIONAL. - **Keys**: - - **serialization_format**: To improve the compactness of the data there are several ways to show to which index a value belongs. + - **serialization_format**: To improve the compactness of the data, there are several ways to show to which index a value belongs. This is specified by the :property:`serialization_format`. - **Type**: string @@ -506,7 +506,7 @@ Ranged Properties - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. - **range_ids**: A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range. - For example, when data of a MD trajectory is shared it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. i.e. which energy belongs to which set of cartesian_site_positions. + For example, when data of an MD trajectory is shared, it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. i.e. which energy belongs to which set of cartesian_site_positions. - **Type**: list of strings - **Requirements/Conventions**: OPTIONAL @@ -528,11 +528,11 @@ Ranged Properties The default value is 0 in each dimension. - **step_size_linear**: If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. - e.g. If the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8 . + e.g. If the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8. - **Type**: list of float - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "linear". - Otherwise it SHOULD NOT be present. + Otherwise, it SHOULD NOT be present. - **offset_regular**: If :property:`serialization_format` is set to :val:`"regular"` this property gives the indexes of the first value. @@ -544,13 +544,13 @@ Ranged Properties - **Type**: list of integers - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. - Otherwise it SHOULD NOT be present. + Otherwise, it SHOULD NOT be present. - **indexes**: If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. - **Type**: List of lists of integers - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "custom". - Otherwise it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. + Otherwise, it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. - **Example**: @@ -1013,6 +1013,7 @@ Standard OPTIONAL URL query parameters standardized by the JSON API specificatio The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON API 1.0 specification. The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. An example is shown in section `Entry Listing Info Endpoints`_. + - **include**: A server MAY implement the JSON API concept of returning `compound documents `__ by utilizing the :query-param:`include` query parameter as specified by `JSON API 1.0 `__. All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. @@ -1038,19 +1039,24 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` -- **property\_ranges**: Specifies which ranges should be returned for ranged properties. +- **property\_ranges**: specifies which ranges should be returned for ranged properties. + It MUST be supported by databases having ranged properties. It consists of a property name directly followed by the range that should be returned. - Ranges can be specified for multiple properties by separating them with a comma. - The ranges are 1 based. i.e. The first value is value 1. + A range is written as a list with a list for each dimension. + Each dimensions list has three values. The first value of the range specifies the first index in that dimension for which values should be returned. - The second value specifies the last index for which values should be be returned. + The second value specifies the last index for which values should be returned. The third value specifies the step size. - Databases that use ranged properties SHOULD support this query parameter. + Ranges can be specified for multiple properties by separating them with a comma. + Databases MUST return the values belonging to properties listed and SHOULD use the ranges in this query parameter. + For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned. + For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`. + The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for a the range :val:`[10,20,1]` the last value returned belongs to index 20. Example: - If there would be a structure with id: id_ property :ranged-property:`_ranged_test_field` with the values :val:`[[9.64, 7.52, 0.69, 5.69],[4.82, 8.35, 3.26, 3.25],[4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2],[2, 3, 1]]` - will return the value: :val:`[[7.52, 0.69],[2.78, 7.87]]` - Multiple ranges can be requested in one query. e.g. :query-paramproperty_ranges=_ranged_test_field[[1, 3, 2],[2, 3, 1]], _ranged_other_field[]` + If there would be a structure with id: id_12345 and a property :ranged-property:`_ranged_test_field` with the values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]]` + will return the value: :val:`[[7.52, 0.69], [2.78, 7.87]]` + Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]], _ranged_other_field[[1,100,1]]` Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". From 16aba076f8bd6926cd83acc7dad37bc508de2555 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 12 Jan 2023 12:44:09 +0100 Subject: [PATCH 06/30] changed description field --- optimade.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index 953b44276..51fb8a563 100644 --- a/optimade.rst +++ b/optimade.rst @@ -452,9 +452,9 @@ Ranged Properties - **Description**: Ranged properties support slicing, so the client can request that only some of the values need to be returned. Likewise, the server can use paging to return the property in multiple parts. This can be useful for properties that are so large that it can be inconvenient to return them in a single response. - In that case a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. - They also provide a method to correlate the values of two ranged properties. - The metadata is returned by default, the values are only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. + If an entry is too large to be returned in a single response a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. + Ranged properties also provide a method to correlate the values of two ranged properties via a :property:`range_ids`. + The metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. - **Type**: dictionary with keys: @@ -474,7 +474,7 @@ Ranged Properties - **Support**: OPTIONAL support in implementations. - Ranged properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first. - - If the part of the property name after the `_ranged_` prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. + - If the part of the property name after the "_ranged_" prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. - By default, only the metadata SHOULD be returned (i.e. all the fields except :property:`values` and :property:`indexes`). The :property:`values` and :property:`indexes` fields SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. From b1d69a873323b0cb3ac8954b078e039567e106c6 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 12 Jan 2023 15:20:29 +0100 Subject: [PATCH 07/30] Apply suggestions from code review Added suggestion from @rartino Co-authored-by: Rickard Armiento --- optimade.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/optimade.rst b/optimade.rst index 51fb8a563..96d1cab2f 100644 --- a/optimade.rst +++ b/optimade.rst @@ -456,6 +456,7 @@ Ranged Properties Ranged properties also provide a method to correlate the values of two ranged properties via a :property:`range_ids`. The metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. +When a client does not use query parameters to select a range for the ranged property, the server returns a dictionary with meta data about the range property with the following format: - **Type**: dictionary with keys: - :property:`serialization_format`: string (REQUIRED) From 14de45d63f5cbc0d090f52006b6220ebaa840489 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 17 Jan 2023 15:11:36 +0100 Subject: [PATCH 08/30] Apply suggestions from code review Vaitkus Co-authored-by: Antanas Vaitkus --- optimade.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index 270caa8d0..a1f577a00 100644 --- a/optimade.rst +++ b/optimade.rst @@ -570,7 +570,7 @@ When a client does not use query parameters to select a range for the ranged pro [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]], // ... - ], + ] }, "_ranged_species_at_sites": { "n_dim": 1, @@ -580,14 +580,14 @@ When a client does not use query parameters to select a range for the ranged pro "offset_regular": [0], "step_size_regular": [1], "nvalues": 3, - "values": ["He", "Ne", "Ar"], + "values": ["He", "Ne", "Ar"] }, "_exmpl_ranged_time":{ "n_dim": 1, "dim_size": [100], "range_ids": ["mdsteps"], "serialization_format": "linear", - "step_size_linear": 0.2, + "step_size_linear": 0.2 }, "_exmpl_ranged_thermostat": { "n_dim": 1, @@ -595,7 +595,7 @@ When a client does not use query parameters to select a range for the ranged pro "range_ids": ["mdsteps"], "serialization_format": "custom", "nvalues": 3, - "values": [20, 40, 60] + "values": [20, 40, 60], "indexes": [[0], [20], [80]] } } @@ -1052,7 +1052,7 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: Databases MUST return the values belonging to properties listed and SHOULD use the ranges in this query parameter. For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned. For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`. - The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for a the range :val:`[10,20,1]` the last value returned belongs to index 20. + The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`[10,20,1]` the last value returned belongs to index 20. Example: If there would be a structure with id: id_12345 and a property :ranged-property:`_ranged_test_field` with the values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]]` From 906db815e97f7d3dc1b12538ca4a93a7e2220238 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 17 Jan 2023 15:17:02 +0100 Subject: [PATCH 09/30] intermediate state from implementing code review. --- optimade.rst | 229 +++++++++++++++++++++++++-------------------------- 1 file changed, 110 insertions(+), 119 deletions(-) diff --git a/optimade.rst b/optimade.rst index 51fb8a563..0b34121cb 100644 --- a/optimade.rst +++ b/optimade.rst @@ -449,26 +449,13 @@ For example, the following query can be sent to API implementations `exmpl1` and Ranged Properties ----------------- -- **Description**: Ranged properties support slicing, so the client can request that only some of the values need to be returned. - Likewise, the server can use paging to return the property in multiple parts. - This can be useful for properties that are so large that it can be inconvenient to return them in a single response. - If an entry is too large to be returned in a single response a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. - Ranged properties also provide a method to correlate the values of two ranged properties via a :property:`range_ids`. - The metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. - -- **Type**: dictionary with keys: - - - :property:`serialization_format`: string (REQUIRED) - - :property:`values`: List of any data type (OPTIONAL) - - :property:`nvalues`: integer (OPTIONAL) - - :property:`range_ids`: list of strings (OPTIONAL) - - :property:`n_dim`: integer (REQUIRED) - - :property:`dim_size`: list of integers (REQUIRED) - - :property:`offset_linear`: float (OPTIONAL) - - :property:`step_size_linear`: list of floats (OPTIONAL) - - :property:`offset_regular`: list of integers (OPTIONAL) - - :property:`step_size_regular`: list of integers (OPTIONAL) - - :property:`indexes`: list of lists of integers (OPTIONAL) +Ranged properties support slicing, so the client can request that only some of the values need to be returned. +Likewise, the server can also use slicing to reduce the size of the response and return the property in multiple parts. +This can be useful for entries/properties that are so large that it can be inconvenient to return them in a single response. +If an entry is too large to be returned in a single response a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. +Multiple ranged properties in the same entry can either be controlled by separate independent ranges or be "correlated" in the sense of having the same range. +The metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. +When a client does not use query parameters to select a range for the ranged property, the server returns a dictionary with metadata about the ranged property with the following format: - **Requirements/Conventions**: @@ -479,127 +466,82 @@ Ranged Properties The :property:`values` and :property:`indexes` fields SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. - **Query**: Queries on the dictionary fields SHOULD be supported, except for the :property:`values` and :property:`indexes` fields for which querying is OPTIONAL. - - **Keys**: - **serialization_format**: To improve the compactness of the data, there are several ways to show to which index a value belongs. This is specified by the :property:`serialization_format`. - - **Type**: string - - **Requirements/Conventions**: This field MUST be present. - - **Possible values**: +The dictionary MUST include these fields: - - **linear**: The value is a linear function of the indexes. - This function is defined by :property:`offset_linear` and :property:`step_size_linear`. - - **regular**: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` indicating the index of the first value. - - **custom**: A separate list with indexes is defined in the field :property:`indexes` to indicate to which index each value belongs. +- :field:`serialization_format`: string. + To improve the compactness of the data there are several ways to show to which index a value belongs. + The string MUST take one of the following values: - - **values**: The values belonging to this property. - The format of this field depends on the property for which data is stored. + - `linear`: The value is a linear function of the indexes. + This function is defined by :property:`offset_linear` and :property:`step_size_linear`. + - `regular`: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` indicating the index of the first value. + - `custom`: A separate list with indexes is defined in the field :property:`indexes` to indicate to which index each value belongs. - - **Type**: List of Any - - **Requirements/Conventions**: The property :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. +- :field:`n_dim`: integer. + The number of dimensions this property has. - - **nvalues**: The number of values in the field :property:`values`. +- :field:`dim_size`: list of integers. + The dimensions of the range in each dimension. - - **Type**: integer - - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. - - **range_ids**: A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range. - For example, when data of an MD trajectory is shared, it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. i.e. which energy belongs to which set of cartesian_site_positions. +Depending on the value of the :property:`serialization_format`, the following fields MUST be present or SHOULD NOT be present in the dictionary: - - **Type**: list of strings - - **Requirements/Conventions**: OPTIONAL +- :field:`nvalues`: integer. + The number of values in the field :property:`values`. + This field MUST be present when :property:`serialization_format` is not set to :val:`"linear"` else it SHOULD NOT be present. - - **n_dim**: The number of dimensions this property has. - - **Type**: integer - - **Requirements/Conventions**: REQUIRED - It MUST be equal to the length of the dim_size field. +- :field:`step_size_linear`:list of floats. + If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. + For example, if the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8. + The value MUST be present when :property:`serialization_format` is set to "linear". + Otherwise, it SHOULD NOT be present. - - **dim_size**: The dimensions of the range in each dimension. +- :field:`step_size_regular`: list of integers. + If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension. + The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. + Otherwise, it SHOULD NOT be present. - - **Type**: list of integers - - **Requirements/Conventions**: REQUIRED - - **offset_linear**: If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. +Depending on the value of the :property:`serialization_format`, the following fields MAY be present or SHOULD NOT be present in the dictionary: - - **Type**: float - - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value SHOULD NOT be present. - The default value is 0 in each dimension. +- :field:`offset_linear`: float. + If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. + The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value SHOULD NOT be present. + The default value is 0. - - **step_size_linear**: If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. - e.g. If the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8. +- :field:`offset_regular`: list of integers. + If :property:`serialization_format` is set to :val:`"regular"` this property gives the indexes of the first value. + The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value SHOULD NOT be present. + The default value is 1 in every dimension. - - **Type**: list of float - - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "linear". - Otherwise, it SHOULD NOT be present. - - **offset_regular**: If :property:`serialization_format` is set to :val:`"regular"` this property gives the indexes of the first value. +The metadata dictionary MAY include these fields: - - **Type**: list of integers - - **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value SHOULD NOT be present. - The default value is 1 in every dimension. +- :field:`range_ids`: list of strings. + A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range. + For example, when data of an MD trajectory is shared, it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. + i.e. which energy belongs to which set of cartesian_site_positions. - - **step_size_regular**: If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension. - - **Type**: list of integers - - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. - Otherwise, it SHOULD NOT be present. +If a ranged property has been included in the query parameter :query-param:`property_ranges` +The following properties MUST be present or SHOULD NOT be present. - - **indexes**: If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. - - - **Type**: List of lists of integers - - **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "custom". - Otherwise, it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. - -- **Example**: - - .. code:: jsonc - - { - "_ranged_cartesian_site_positions": { - "n_dim": 3, - "dim_size": [100, 3, 3], - "range_ids": ["mdsteps","particles","xyz"], - "serialization_format": "regular", - "offset_regular": [1, 1, 1], - "step_size_regular": [1, 1, 1], - "nvalues": 900, - "values": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], - [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], - [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]], - // ... - ], - }, - "_ranged_species_at_sites": { - "n_dim": 1, - "dim_size": [3], - "range_ids": ["particles"], - "serialization_format": "regular", - "offset_regular": [0], - "step_size_regular": [1], - "nvalues": 3, - "values": ["He", "Ne", "Ar"], - }, - "_exmpl_ranged_time":{ - "n_dim": 1, - "dim_size": [100], - "range_ids": ["mdsteps"], - "serialization_format": "linear", - "step_size_linear": 0.2, - }, - "_exmpl_ranged_thermostat": { - "n_dim": 1, - "dim_size": [100], - "range_ids": ["mdsteps"], - "serialization_format": "custom", - "nvalues": 3, - "values": [20, 40, 60] - "indexes": [[0], [20], [80]] - } - } +- :field:`values`: List of Any + The values belonging to this property. + The format of this field depends on the property for which data is stored. + The property :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. +- :field:`indexes`: List of lists of integers + If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. + The value MUST be present when :property:`serialization_format` is set to "custom". + Otherwise, it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. +- :field:`returned range`: Responses ========= @@ -856,6 +798,54 @@ An example of a full response: ] } + +- Several examples of how ranged properties can be returned in the JSON format: + + .. code:: jsonc + + { + "_ranged_cartesian_site_positions": { + "n_dim": 3, + "dim_size": [100, 3, 3], + "range_ids": ["mdsteps","particles","xyz"], + "serialization_format": "regular", + "offset_regular": [1, 1, 1], + "step_size_regular": [1, 1, 1], + "nvalues": 900, + "values": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], + [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], + [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]], + // ... + ], + }, + "_ranged_species_at_sites": { + "n_dim": 1, + "dim_size": [3], + "range_ids": ["particles"], + "serialization_format": "regular", + "offset_regular": [0], + "step_size_regular": [1], + "nvalues": 3, + "values": ["He", "Ne", "Ar"], + }, + "_exmpl_ranged_time":{ + "n_dim": 1, + "dim_size": [100], + "range_ids": ["mdsteps"], + "serialization_format": "linear", + "step_size_linear": 0.2, + }, + "_exmpl_ranged_thermostat": { + "n_dim": 1, + "dim_size": [100], + "range_ids": ["mdsteps"], + "serialization_format": "custom", + "nvalues": 3, + "values": [20, 40, 60] + "indexes": [[0], [20], [80]] + } + } + HTTP Response Status Codes -------------------------- @@ -1039,16 +1029,17 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` -- **property\_ranges**: specifies which ranges should be returned for ranged properties. +- **property\_ranges**: specifies which data ranges should be returned for ranged properties. It MUST be supported by databases having ranged properties. It consists of a property name directly followed by the range that should be returned. - A range is written as a list with a list for each dimension. - Each dimensions list has three values. + A range is a list containing a list for each dimension. + Each dimensions list has three integer values. The first value of the range specifies the first index in that dimension for which values should be returned. The second value specifies the last index for which values should be returned. The third value specifies the step size. + A list consists of a pair of square brackets ("[", ASCII 91(0x5B)) and ("]", ASCII 93(0x5D)) enclosing a number of values separated by a comma (",", ASCII 91(0x5B)) Ranges can be specified for multiple properties by separating them with a comma. - Databases MUST return the values belonging to properties listed and SHOULD use the ranges in this query parameter. + Databases MUST return the :property:`values` and :property:`indexes` field belonging to properties listed and SHOULD use the ranges in this query parameter. For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned. For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`. The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for a the range :val:`[10,20,1]` the last value returned belongs to index 20. From a96dffe6af36a3a389410599c8e40c01f5f46510 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 18 Jan 2023 21:38:32 +0100 Subject: [PATCH 10/30] Processed comments rartino. --- optimade.rst | 96 +++++++++++++++++++++++++--------------------------- 1 file changed, 46 insertions(+), 50 deletions(-) diff --git a/optimade.rst b/optimade.rst index 90f44b2ec..a65d18b9c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -520,7 +520,7 @@ Depending on the value of the :property:`serialization_format`, the following fi The default value is 1 in every dimension. -The metadata dictionary MAY include these fields: +The dictionary MAY include these fields: - :field:`range_ids`: list of strings. A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range. @@ -529,7 +529,7 @@ The metadata dictionary MAY include these fields: If a ranged property has been included in the query parameter :query-param:`property_ranges` -The following properties MUST be present or SHOULD NOT be present. +The following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. - :field:`values`: List of Any The values belonging to this property. @@ -541,54 +541,6 @@ The following properties MUST be present or SHOULD NOT be present. The value MUST be present when :property:`serialization_format` is set to "custom". Otherwise, it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. -- **Example**: - - .. code:: jsonc - - { - "_ranged_cartesian_site_positions": { - "n_dim": 3, - "dim_size": [100, 3, 3], - "range_ids": ["mdsteps","particles","xyz"], - "serialization_format": "regular", - "offset_regular": [1, 1, 1], - "step_size_regular": [1, 1, 1], - "nvalues": 900, - "values": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], - [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], - [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]], - // ... - ] - }, - "_ranged_species_at_sites": { - "n_dim": 1, - "dim_size": [3], - "range_ids": ["particles"], - "serialization_format": "regular", - "offset_regular": [0], - "step_size_regular": [1], - "nvalues": 3, - "values": ["He", "Ne", "Ar"] - }, - "_exmpl_ranged_time":{ - "n_dim": 1, - "dim_size": [100], - "range_ids": ["mdsteps"], - "serialization_format": "linear", - "step_size_linear": 0.2 - }, - "_exmpl_ranged_thermostat": { - "n_dim": 1, - "dim_size": [100], - "range_ids": ["mdsteps"], - "serialization_format": "custom", - "nvalues": 3, - "values": [20, 40, 60], - "indexes": [[0], [20], [80]] - } - } - - Responses ========= @@ -848,6 +800,50 @@ An example of a full response: - Several examples of how ranged properties can be returned in the JSON format: + .. code:: jsonc + + { + "_ranged_cartesian_site_positions": { + "n_dim": 3, + "dim_size": [100, 3, 3], + "range_ids": ["mdsteps","particles","xyz"], + "serialization_format": "regular", + "offset_regular": [1, 1, 1], + "step_size_regular": [1, 1, 1], + "nvalues": 900, + "values": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], + [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], + [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]], + // ... + ] + }, + "_ranged_species_at_sites": { + "n_dim": 1, + "dim_size": [3], + "range_ids": ["particles"], + "serialization_format": "regular", + "offset_regular": [0], + "step_size_regular": [1], + "nvalues": 3, + "values": ["He", "Ne", "Ar"] + }, + "_exmpl_ranged_time":{ + "n_dim": 1, + "dim_size": [100], + "range_ids": ["mdsteps"], + "serialization_format": "linear", + "step_size_linear": 0.2 + }, + "_exmpl_ranged_thermostat": { + "n_dim": 1, + "dim_size": [100], + "range_ids": ["mdsteps"], + "serialization_format": "custom", + "nvalues": 3, + "values": [20, 40, 60], + "indexes": [[0], [20], [80]] + } + } HTTP Response Status Codes From 15f599cc4815d513f82c230f2cbcef05bc34d12f Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 15 Feb 2023 13:33:09 +0100 Subject: [PATCH 11/30] Small corrections. --- optimade.rst | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index a65d18b9c..73118d247 100644 --- a/optimade.rst +++ b/optimade.rst @@ -453,9 +453,8 @@ Ranged properties support slicing, so the client can request that only some of t Likewise, the server can also use slicing to reduce the size of the response and return the property in multiple parts. This can be useful for entries/properties that are so large that it can be inconvenient to return them in a single response. If an entry is too large to be returned in a single response a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. -Multiple ranged properties in the same entry can either be controlled by separate independent ranges or be "correlated" in the sense of having the same range. +Ranged properties in the same entry can have separate independent ranges, or the same range so they are "correlated" e.g. if an energy and a set of particle positions have the same index this energy belongs to those particle positions. The metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. -When a client does not use query parameters to select a range for the ranged property, the server returns a dictionary with metadata about the ranged property with the following format: - **Requirements/Conventions**: @@ -467,10 +466,6 @@ When a client does not use query parameters to select a range for the ranged pro - **Query**: Queries on the dictionary fields SHOULD be supported, except for the :property:`values` and :property:`indexes` fields for which querying is OPTIONAL. - - **serialization_format**: To improve the compactness of the data, there are several ways to show to which index a value belongs. - This is specified by the :property:`serialization_format`. - - The dictionary MUST include these fields: - :field:`serialization_format`: string. From 73905dc66b8d03fac276f8858c88aa8e075b27f9 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 15 Feb 2023 17:45:52 +0100 Subject: [PATCH 12/30] Added returned range property. --- optimade.rst | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 73118d247..1edbb98aa 100644 --- a/optimade.rst +++ b/optimade.rst @@ -536,6 +536,14 @@ The following properties MUST be present or SHOULD NOT be present, depending on The value MUST be present when :property:`serialization_format` is set to "custom". Otherwise, it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. +- :field:`returned_range`: List of list of integers + The range belonging to the returned data. It uses the same format as the property_ranges query param. + It consist of a list which for each dimension contains a list of three values. + The first value indicates the index of the first value that has been returned. + The second value indicates the index of last returned value. + The third value is the step size. + It is only returned when the `serialization_format`is not "linear". + Responses ========= @@ -808,9 +816,10 @@ An example of a full response: "nvalues": 900, "values": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], - [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]], + [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]] // ... ] + "returned_range": [[1,100,2],[1,3,1],[1,3,1]] }, "_ranged_species_at_sites": { "n_dim": 1, @@ -821,6 +830,7 @@ An example of a full response: "step_size_regular": [1], "nvalues": 3, "values": ["He", "Ne", "Ar"] + "returned_range":[[1,3,1]] }, "_exmpl_ranged_time":{ "n_dim": 1, @@ -1040,7 +1050,7 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`[10,20,1]` the last value returned belongs to index 20. Example: - If there would be a structure with id: id_12345 and a property :ranged-property:`_ranged_test_field` with the values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]]` + If there would be a structure with id: id_12345 and a property :ranged-property:`_ranged_test_field` with the values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]]` will return the value: :val:`[[7.52, 0.69], [2.78, 7.87]]` Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]], _ranged_other_field[[1,100,1]]` From c6834f3dc76e4f8dddb44efca295df62f5aa6476 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 16 Feb 2023 18:51:16 +0100 Subject: [PATCH 13/30] Added extra explanation values field. --- optimade.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index 1edbb98aa..b15cc1274 100644 --- a/optimade.rst +++ b/optimade.rst @@ -454,7 +454,7 @@ Likewise, the server can also use slicing to reduce the size of the response and This can be useful for entries/properties that are so large that it can be inconvenient to return them in a single response. If an entry is too large to be returned in a single response a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. Ranged properties in the same entry can have separate independent ranges, or the same range so they are "correlated" e.g. if an energy and a set of particle positions have the same index this energy belongs to those particle positions. -The metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. +Only the metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. - **Requirements/Conventions**: @@ -523,13 +523,13 @@ The dictionary MAY include these fields: i.e. which energy belongs to which set of cartesian_site_positions. -If a ranged property has been included in the query parameter :query-param:`property_ranges` -The following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. +If a ranged property has been included in the query parameter :query-param:`property_ranges`, the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. - :field:`values`: List of Any The values belonging to this property. + The range for which these values are returned is specified by the :query-param:`property_ranges` as described in `Entry Listing URL Query Parameters`_. The format of this field depends on the property for which data is stored. - The property :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. + The property :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. - :field:`indexes`: List of lists of integers If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. @@ -539,10 +539,10 @@ The following properties MUST be present or SHOULD NOT be present, depending on - :field:`returned_range`: List of list of integers The range belonging to the returned data. It uses the same format as the property_ranges query param. It consist of a list which for each dimension contains a list of three values. - The first value indicates the index of the first value that has been returned. + The first value indicates the index, in that dimension, of the first value that has been returned. The second value indicates the index of last returned value. The third value is the step size. - It is only returned when the `serialization_format`is not "linear". + It is only returned when the `serialization_format` is not "linear". Responses From b0cc94cbe290f452d3d5a19642de51f1599a5b0c Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 17 Feb 2023 10:55:23 +0100 Subject: [PATCH 14/30] Apply suggestions from code review Co-authored-by: Rickard Armiento --- optimade.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index b15cc1274..ae33e7d8c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1038,21 +1038,24 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: It MUST be supported by databases having ranged properties. It consists of a property name directly followed by the range that should be returned. A range is a list containing a list for each dimension. - Each dimensions list has three integer values. + Each dimension's list has three integer values. The first value of the range specifies the first index in that dimension for which values should be returned. The second value specifies the last index for which values should be returned. The third value specifies the step size. A list consists of a pair of square brackets ("[", ASCII 91(0x5B)) and ("]", ASCII 93(0x5D)) enclosing a number of values separated by a comma (",", ASCII 91(0x5B)) Ranges can be specified for multiple properties by separating them with a comma. + The field may include optional space characters, which do no alter the meaning of the expression. Databases MUST return the :property:`values` and :property:`indexes` field belonging to properties listed and SHOULD use the ranges in this query parameter. For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned. For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`. The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`[10,20,1]` the last value returned belongs to index 20. Example: - If there would be a structure with id: id_12345 and a property :ranged-property:`_ranged_test_field` with the values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]]` - will return the value: :val:`[[7.52, 0.69], [2.78, 7.87]]` - Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]], _ranged_other_field[[1,100,1]]` + A database has a :entry:`structure` entry with id: :val:`id_12345` and a ranged property :property:`_ranged_test_field` with the two-dimensional data values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]]`. + A client makes a request :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]]`. + The response is then a single entry response for structure `12345` where the `_ranged_test_field` property is included with the `value` key set to :val:`[[7.52, 0.69], [2.78, 7.87]]`. + + Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]], _ranged_other_field[[1,100,1]]`. Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". From d7c8a9ce12e7a348d8e75eae74abc76ba3939494 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 9 Mar 2023 13:05:23 +0100 Subject: [PATCH 15/30] Processed comments rickard and a few more small improvements. --- optimade.rst | 121 +++++++++++++++++++++++++++------------------------ 1 file changed, 63 insertions(+), 58 deletions(-) diff --git a/optimade.rst b/optimade.rst index 62e867005..1f2f231c7 100644 --- a/optimade.rst +++ b/optimade.rst @@ -449,24 +449,19 @@ For example, the following query can be sent to API implementations `exmpl1` and Ranged Properties ----------------- -Ranged properties support slicing, so the client can request that only some of the values need to be returned. -Likewise, the server can also use slicing to reduce the size of the response and return the property in multiple parts. -This can be useful for entries/properties that are so large that it can be inconvenient to return them in a single response. -If an entry is too large to be returned in a single response a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved. -Ranged properties in the same entry can have separate independent ranges, or the same range so they are "correlated" e.g. if an energy and a set of particle positions have the same index this energy belongs to those particle positions. -Only the metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_. +Ranged properties are used for properties that are too large to be returned by default for every entry in a response. They can also support slicing, so the client can request that only some of the values need to be returned. The server can limit the size of the response and offer the values of the property in multiple parts. + - **Requirements/Conventions**: - **Support**: OPTIONAL support in implementations. - - Ranged properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first. - - If the part of the property name after the "_ranged_" prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property. - - By default, only the metadata SHOULD be returned (i.e. all the fields except :property:`values` and :property:`indexes`). - The :property:`values` and :property:`indexes` fields SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_. + - A ranged property can be recognized by the presence of the :field:`range` in the metadata of the property, i.e. in the field: _meta. + - The server may return null or only part of the values of the property under the the field :field:``. In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. - - **Query**: Queries on the dictionary fields SHOULD be supported, except for the :property:`values` and :property:`indexes` fields for which querying is OPTIONAL. + - **Query**: Support for querying on the values of a ranged field is OPTIONAL, even when the non-ranged version of this field has stricter requirements on the query-ability. This is because the ranged version of a property can be much larger, making queries on the value very slow. -The dictionary MUST include these fields: +The dictionary under :field:`_meta.range` MUST include these fields. +All fields in this section SHOULD be queryable with support for all mandatory filter features. - :field:`serialization_format`: string. To improve the compactness of the data there are several ways to show to which index a value belongs. @@ -477,20 +472,22 @@ The dictionary MUST include these fields: - `regular`: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` indicating the index of the first value. - `custom`: A separate list with indexes is defined in the field :property:`indexes` to indicate to which index each value belongs. -- :field:`n_dim`: integer. - The number of dimensions this property has. -- :field:`dim_size`: list of integers. - The dimensions of the range in each dimension. +- :field:`n_indexable_dim`: integer. + The number of dimensions that can be indexed. The values themselves can also be lists, so a property may have more dimensions than listed here, but it may not be practical for the server to support indexing at that level. -Depending on the value of the :property:`serialization_format`, the following fields MUST be present or SHOULD NOT be present in the dictionary: +- :field:`dim_size`: list of integers. + The size of the range in each indexable dimension. + +Depending on the value of the :field:`serialization_format`, the following fields MUST be present or SHOULD NOT be present in the dictionary. +All fields in this section SHOULD be queryable with support for all mandatory filter features. - :field:`nvalues`: integer. The number of values in the field :property:`values`. This field MUST be present when :property:`serialization_format` is not set to :val:`"linear"` else it SHOULD NOT be present. -- :field:`step_size_linear`:list of floats. +- :field:`step_size_linear`: list of floats. If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. For example, if the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8. The value MUST be present when :property:`serialization_format` is set to "linear". @@ -502,7 +499,8 @@ Depending on the value of the :property:`serialization_format`, the following fi Otherwise, it SHOULD NOT be present. -Depending on the value of the :property:`serialization_format`, the following fields MAY be present or SHOULD NOT be present in the dictionary: +Depending on the value of the :property:`serialization_format`, the following fields MAY be present or SHOULD NOT be present in the dictionary. +All fields in this section SHOULD be queryable with support for all mandatory filter features. - :field:`offset_linear`: float. If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. @@ -514,35 +512,36 @@ Depending on the value of the :property:`serialization_format`, the following fi The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value SHOULD NOT be present. The default value is 1 in every dimension. - -The dictionary MAY include these fields: +The dictionary MAY include these fields. - :field:`range_ids`: list of strings. A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range. For example, when data of an MD trajectory is shared, it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. i.e. which energy belongs to which set of cartesian_site_positions. + SHOULD be a queryable property with support for all mandatory filter features. - -If a ranged property has been included in the query parameter :query-param:`property_ranges`, the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. - -- :field:`values`: List of Any - The values belonging to this property. - The range for which these values are returned is specified by the :query-param:`property_ranges` as described in `Entry Listing URL Query Parameters`_. - The format of this field depends on the property for which data is stored. - The property :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`. +If the :field:`` contains data the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. Querying is not relevant for these properties and SHOULD NOT be supported. - :field:`indexes`: List of lists of integers - If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong. + If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the field :field:`` belong. The value MUST be present when :property:`serialization_format` is set to "custom". Otherwise, it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. - :field:`returned_range`: List of list of integers - The range belonging to the returned data. It uses the same format as the property_ranges query param. + The range belonging to the returned data. It uses the same format as the :query-param:`property_ranges` query paramameter. It consist of a list which for each dimension contains a list of three values. The first value indicates the index, in that dimension, of the first value that has been returned. The second value indicates the index of last returned value. The third value is the step size. - It is only returned when the `serialization_format` is not "linear". + It MUST be returned when the `serialization_format` is "regular". Otherwise it should not be returned. + +- :field:`next`: a `JSON API links object `__ + If the field is too large to be returned in a single response, this field will contain a URL pointing to the entry, to which this property belongs, which contains the next set of values for this property. + If all the data for this property has been returned its value should be `null` + +- :field:`more_data_available`: :field-val:`false` if all the values in the requested range have been returned and :field-val:`true` if the returned values are incomplete. + + Responses @@ -803,26 +802,31 @@ An example of a full response: - Several examples of how ranged properties can be returned in the JSON format: - .. code:: jsonc +.. code:: jsonc - { - "_ranged_cartesian_site_positions": { - "n_dim": 3, - "dim_size": [100, 3, 3], - "range_ids": ["mdsteps","particles","xyz"], - "serialization_format": "regular", - "offset_regular": [1, 1, 1], - "step_size_regular": [1, 1, 1], - "nvalues": 900, - "values": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], - [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], - [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]] - // ... - ] - "returned_range": [[1,100,2],[1,3,1],[1,3,1]] + { + "cartesian_site_positions": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], + [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], + [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]] + // ... + ], + "cartesian_site_positions_meta": { + "range { + "n_indexable_dim": 3, + "dim_size": [100, 3, 3], + "range_ids": ["mdsteps","particles","xyz"], + "serialization_format": "regular", + "offset_regular": [1, 1, 1], + "step_size_regular": [1, 1, 1], + "nvalues": 900, + "returned¨_range": [[1,100,2],[1,3,1],[1,3,1]] + "more_data_available": true + "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=cartesian_site positions[[101,900,2],[1,3,1],[1,3,1]] + } }, - "_ranged_species_at_sites": { - "n_dim": 1, + "species_at_sites": ["He", "Ne", "Ar"], + + "n_indexable_dim": 1, "dim_size": [3], "range_ids": ["particles"], "serialization_format": "regular", @@ -833,14 +837,14 @@ An example of a full response: "returned_range":[[1,3,1]] }, "_exmpl_ranged_time":{ - "n_dim": 1, + "n_indexable_dim": 1, "dim_size": [100], "range_ids": ["mdsteps"], "serialization_format": "linear", "step_size_linear": 0.2 }, "_exmpl_ranged_thermostat": { - "n_dim": 1, + "n_indexable_dim": 1, "dim_size": [100], "range_ids": ["mdsteps"], "serialization_format": "custom", @@ -1035,27 +1039,28 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` - **property\_ranges**: specifies which data ranges should be returned for ranged properties. - It MUST be supported by databases having ranged properties. + In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. It consists of a property name directly followed by the range that should be returned. A range is a list containing a list for each dimension. + A list consists of a pair of square brackets ("[", ASCII 91(0x5B)) and ("]", ASCII 93(0x5D)) enclosing a number of values separated by commas (",", ASCII 91(0x5B)) Each dimension's list has three integer values. The first value of the range specifies the first index in that dimension for which values should be returned. The second value specifies the last index for which values should be returned. The third value specifies the step size. - A list consists of a pair of square brackets ("[", ASCII 91(0x5B)) and ("]", ASCII 93(0x5D)) enclosing a number of values separated by a comma (",", ASCII 91(0x5B)) + Ranges can be specified for multiple properties by separating them with a comma. The field may include optional space characters, which do no alter the meaning of the expression. - Databases MUST return the :property:`values` and :property:`indexes` field belonging to properties listed and SHOULD use the ranges in this query parameter. + Databases SHOULD return the values falling within this range and the :property:`indexes` or :property:`returned_range` field belonging to properties listed. For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned. For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`. The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`[10,20,1]` the last value returned belongs to index 20. Example: A database has a :entry:`structure` entry with id: :val:`id_12345` and a ranged property :property:`_ranged_test_field` with the two-dimensional data values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]]`. - A client makes a request :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]]`. - The response is then a single entry response for structure `12345` where the `_ranged_test_field` property is included with the `value` key set to :val:`[[7.52, 0.69], [2.78, 7.87]]`. + A client makes a request :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=test_field[[1, 3, 2], [2, 3, 1]]`. + The response is then a single entry response for structure `12345` where the `test_field` property is included with the `value` key set to :val:`[[7.52, 0.69], [2.78, 7.87]]`. - Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]], _ranged_other_field[[1,100,1]]`. + Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=test_field[[1, 3, 2], [2, 3, 1]], other_field[[1,100,1]]`. Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". From d1e8d74e69535558b2afb743befdc4ed22b569a5 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 9 Mar 2023 17:06:42 +0100 Subject: [PATCH 16/30] Further changes after proof reading. --- optimade.rst | 177 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 103 insertions(+), 74 deletions(-) diff --git a/optimade.rst b/optimade.rst index 1f2f231c7..3fb7e6194 100644 --- a/optimade.rst +++ b/optimade.rst @@ -455,15 +455,14 @@ Ranged properties are used for properties that are too large to be returned by d - **Requirements/Conventions**: - **Support**: OPTIONAL support in implementations. - - A ranged property can be recognized by the presence of the :field:`range` in the metadata of the property, i.e. in the field: _meta. - - The server may return null or only part of the values of the property under the the field :field:``. In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. - - - **Query**: Support for querying on the values of a ranged field is OPTIONAL, even when the non-ranged version of this field has stricter requirements on the query-ability. This is because the ranged version of a property can be much larger, making queries on the value very slow. + - A ranged property can be recognized by the presence of the field :field:`range` in the metadata of the property, i.e. in the field: :field:`_meta`. + - The server may return null or only a part of the values of the property under the the field :field:``. In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. The dictionary under :field:`_meta.range` MUST include these fields. All fields in this section SHOULD be queryable with support for all mandatory filter features. - :field:`serialization_format`: string. + To improve the compactness of the data there are several ways to show to which index a value belongs. The string MUST take one of the following values: @@ -474,26 +473,29 @@ All fields in this section SHOULD be queryable with support for all mandatory fi - :field:`n_indexable_dim`: integer. - The number of dimensions that can be indexed. The values themselves can also be lists, so a property may have more dimensions than listed here, but it may not be practical for the server to support indexing at that level. + The number of dimensions that can be indexed. The values themselves can also be lists, so a property may have more dimensions than listed here, but it may not be practical for the server to support indexing at that level. - :field:`dim_size`: list of integers. + The size of the range in each indexable dimension. -Depending on the value of the :field:`serialization_format`, the following fields MUST be present or SHOULD NOT be present in the dictionary. +Depending on the value of the :field:`serialization_format`, the following fields SHOULD/MUST be present or SHOULD NOT be present in the dictionary. All fields in this section SHOULD be queryable with support for all mandatory filter features. -- :field:`nvalues`: integer. - The number of values in the field :property:`values`. - This field MUST be present when :property:`serialization_format` is not set to :val:`"linear"` else it SHOULD NOT be present. +- :field:`n_values`: integer. + + The total number of values in the property. This may be larger than the number of values that are returned. This field SHOULD be present when :property:`serialization_format` is not set to :val:`"linear"` else it SHOULD NOT be present. - :field:`step_size_linear`: list of floats. + If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. For example, if the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8. The value MUST be present when :property:`serialization_format` is set to "linear". Otherwise, it SHOULD NOT be present. - :field:`step_size_regular`: list of integers. + If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension. The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. Otherwise, it SHOULD NOT be present. @@ -503,11 +505,13 @@ Depending on the value of the :property:`serialization_format`, the following fi All fields in this section SHOULD be queryable with support for all mandatory filter features. - :field:`offset_linear`: float. + If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value SHOULD NOT be present. The default value is 0. - :field:`offset_regular`: list of integers. + If :property:`serialization_format` is set to :val:`"regular"` this property gives the indexes of the first value. The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value SHOULD NOT be present. The default value is 1 in every dimension. @@ -515,33 +519,41 @@ All fields in this section SHOULD be queryable with support for all mandatory fi The dictionary MAY include these fields. - :field:`range_ids`: list of strings. - A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range. - For example, when data of an MD trajectory is shared, it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. - i.e. which energy belongs to which set of cartesian_site_positions. + + A list with an identifier for each dimension of the property. If two properties have the same range_id for a dimension it means that the values at an index along this dimension belong to each other. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a trajectory have the same :field:`range_ids` it indicates that the energy at an index x(in the dimension labeled by this range_ids) belongs to the cartesian_site_positions at the same index x. SHOULD be a queryable property with support for all mandatory filter features. If the :field:`` contains data the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. Querying is not relevant for these properties and SHOULD NOT be supported. -- :field:`indexes`: List of lists of integers +- :field:`indexes`: List of lists of integers. + If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the field :field:`` belong. - The value MUST be present when :property:`serialization_format` is set to "custom". - Otherwise, it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`. + The value MUST be present when :property:`serialization_format` is set to :val:`"custom"`. + Otherwise, it SHOULD NOT be present. + +- :field:`n_returned_values`: integer + + The number of values that have been returned. + This value SHOULD be present when `serialization_format` is set to :val:`"custom"` or :val:`"regular"`. Otherwise it SHOULD NOT be present. -- :field:`returned_range`: List of list of integers - The range belonging to the returned data. It uses the same format as the :query-param:`property_ranges` query paramameter. +- :field:`returned_range`: List of list of integers. + + The range belonging to the returned data. It uses the same format as the :query-param:`property_ranges` query parameter. It consist of a list which for each dimension contains a list of three values. The first value indicates the index, in that dimension, of the first value that has been returned. The second value indicates the index of last returned value. The third value is the step size. - It MUST be returned when the `serialization_format` is "regular". Otherwise it should not be returned. + It MUST be returned when the `serialization_format` is "regular" or "custom". Otherwise it should not be returned. -- :field:`next`: a `JSON API links object `__ - If the field is too large to be returned in a single response, this field will contain a URL pointing to the entry, to which this property belongs, which contains the next set of values for this property. - If all the data for this property has been returned its value should be `null` +- :field:`next`: a `JSON API links object `__. -- :field:`more_data_available`: :field-val:`false` if all the values in the requested range have been returned and :field-val:`true` if the returned values are incomplete. + If the field is too large to be returned in a single response, this field will contain a URL pointing to the entry, to which this property belongs, which contains the next set of values for this property. + If all the data for this property has been returned its value should be :val:¨`null` +- :field:`more_data_available`: boolean. + :field-val:`false` if all the values in the requested range have been returned and :field-val:`true` if the returned values are incomplete. Responses @@ -804,55 +816,72 @@ An example of a full response: .. code:: jsonc - { - "cartesian_site_positions": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], - [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], - [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]] - // ... - ], - "cartesian_site_positions_meta": { - "range { - "n_indexable_dim": 3, - "dim_size": [100, 3, 3], - "range_ids": ["mdsteps","particles","xyz"], - "serialization_format": "regular", - "offset_regular": [1, 1, 1], - "step_size_regular": [1, 1, 1], - "nvalues": 900, - "returned¨_range": [[1,100,2],[1,3,1],[1,3,1]] - "more_data_available": true - "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=cartesian_site positions[[101,900,2],[1,3,1],[1,3,1]] - } - }, - "species_at_sites": ["He", "Ne", "Ar"], - - "n_indexable_dim": 1, - "dim_size": [3], - "range_ids": ["particles"], - "serialization_format": "regular", - "offset_regular": [0], - "step_size_regular": [1], - "nvalues": 3, - "values": ["He", "Ne", "Ar"] - "returned_range":[[1,3,1]] - }, - "_exmpl_ranged_time":{ - "n_indexable_dim": 1, - "dim_size": [100], - "range_ids": ["mdsteps"], - "serialization_format": "linear", - "step_size_linear": 0.2 - }, - "_exmpl_ranged_thermostat": { - "n_indexable_dim": 1, - "dim_size": [100], - "range_ids": ["mdsteps"], - "serialization_format": "custom", - "nvalues": 3, - "values": [20, 40, 60], - "indexes": [[0], [20], [80]] - } - } + { + "cartesian_site_positions": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], + [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], + [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]] + // ... + ], + "cartesian_site_positions_meta": { + "range": { + "n_indexable_dim": 3, + "dim_size": [100, 3, 3], + "range_ids": ["mdsteps","particles","xyz"], + "serialization_format": "regular", + "offset_regular": [1, 1, 1], + "step_size_regular": [1, 1, 1], + "n_values": 900, + "n_returned_values": 50, + "returned¨_range": [[1,100,2],[1,3,1],[1,3,1]], + "more_data_available": true, + "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=cartesian_site positions[[101,900,2],[1,3,1],[1,3,1]]" + }, + // ... + }, + "species_at_sites": ["He", "Ne", "Ar"], + "species_at_sites_meta": { + "range": { + "n_indexable_dim": 1, + "dim_size": [3], + "range_ids": ["particles"], + "serialization_format": "regular", + "offset_regular": [1], + "step_size_regular": [1], + "n_returned_values": 3, + "n_values": 3, + "returned_range":[[1,3,1]], + "more_data_available": false, + "next": null + }, + // ... + }, + "_exmpl_ranged_time": null, + "_exmpl_ranged_time_meta":{ + "range":{ + "n_indexable_dim": 1, + "dim_size": [100], + "range_ids": ["mdsteps"], + "serialization_format": "linear", + "step_size_linear": 0.2 + }, + // ... + }, + "_exmpl_ranged_thermostat": [20, 40, 60], + "_exmpl_ranged_thermostat_meta": { + "range": { + "n_indexable_dim": 1, + "dim_size": [100], + "range_ids": ["mdsteps"], + "serialization_format": "custom", + "n_values": 3, + "n_returned_values": 3, + "indexes": [[0], [20], [80]], + "more_data_available": false, + "next": null + }, + // ... + } + } HTTP Response Status Codes @@ -1050,15 +1079,15 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: Ranges can be specified for multiple properties by separating them with a comma. The field may include optional space characters, which do no alter the meaning of the expression. - Databases SHOULD return the values falling within this range and the :property:`indexes` or :property:`returned_range` field belonging to properties listed. + Databases SHOULD return the values falling within this range and the :property:`indexes` or :property:`returned_range` field belonging to the returned values. For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned. For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`. The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`[10,20,1]` the last value returned belongs to index 20. Example: - A database has a :entry:`structure` entry with id: :val:`id_12345` and a ranged property :property:`_ranged_test_field` with the two-dimensional data values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]]`. + A database has a :entry:`structure` entry with id: :val:`id_12345` and a ranged property :property:`test_field` with the two-dimensional data values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]]`. A client makes a request :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=test_field[[1, 3, 2], [2, 3, 1]]`. - The response is then a single entry response for structure `12345` where the `test_field` property is included with the `value` key set to :val:`[[7.52, 0.69], [2.78, 7.87]]`. + The response is then a single entry response for structure `12345` where the `test_field` property is included with the values :val:`[[7.52, 0.69], [2.78, 7.87]]`. Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=test_field[[1, 3, 2], [2, 3, 1]], other_field[[1,100,1]]`. From 139c70ed3619d918be121086ae50a8b642ed234c Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 9 Mar 2023 18:25:24 +0100 Subject: [PATCH 17/30] further refinements. --- optimade.rst | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/optimade.rst b/optimade.rst index 3fb7e6194..e8722399f 100644 --- a/optimade.rst +++ b/optimade.rst @@ -449,21 +449,23 @@ For example, the following query can be sent to API implementations `exmpl1` and Ranged Properties ----------------- -Ranged properties are used for properties that are too large to be returned by default for every entry in a response. They can also support slicing, so the client can request that only some of the values need to be returned. The server can limit the size of the response and offer the values of the property in multiple parts. +Ranged properties are used for properties that are too large to be returned by default for every entry in a response. +They can also support slicing, so the client can request that only some of the values need to be returned. +The server can limit the size of the response and offer the values of the property in multiple parts. - **Requirements/Conventions**: - **Support**: OPTIONAL support in implementations. - A ranged property can be recognized by the presence of the field :field:`range` in the metadata of the property, i.e. in the field: :field:`_meta`. - - The server may return null or only a part of the values of the property under the the field :field:``. In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. + - The server may return null or only a part of the values of the property under the field :field:``. In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. The dictionary under :field:`_meta.range` MUST include these fields. All fields in this section SHOULD be queryable with support for all mandatory filter features. - :field:`serialization_format`: string. - To improve the compactness of the data there are several ways to show to which index a value belongs. + To improve the compactness of the data, there are several ways to show to which index a value belongs. The string MUST take one of the following values: - `linear`: The value is a linear function of the indexes. @@ -491,7 +493,7 @@ All fields in this section SHOULD be queryable with support for all mandatory fi If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. For example, if the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8. - The value MUST be present when :property:`serialization_format` is set to "linear". + The value MUST be present when :property:`serialization_format` is set to :val:`"linear"`. Otherwise, it SHOULD NOT be present. - :field:`step_size_regular`: list of integers. @@ -520,11 +522,12 @@ The dictionary MAY include these fields. - :field:`range_ids`: list of strings. - A list with an identifier for each dimension of the property. If two properties have the same range_id for a dimension it means that the values at an index along this dimension belong to each other. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a trajectory have the same :field:`range_ids` it indicates that the energy at an index x(in the dimension labeled by this range_ids) belongs to the cartesian_site_positions at the same index x. + A list with an identifier for each dimension of the property. + If two properties have the same range_id for a dimension, it means that the values at an index along this dimension belong to each other. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a trajectory have the same :field:`range_ids` it indicates that the energy at an index x(in the dimension labelled by this range_ids) belongs to the cartesian_site_positions at the same index x. SHOULD be a queryable property with support for all mandatory filter features. -If the :field:`` contains data the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. Querying is not relevant for these properties and SHOULD NOT be supported. +If the :field:`` contains data, the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. Querying is not relevant for these properties and SHOULD NOT be supported. - :field:`indexes`: List of lists of integers. @@ -540,20 +543,20 @@ If the :field:`` contains data the following properties MUST be p - :field:`returned_range`: List of list of integers. The range belonging to the returned data. It uses the same format as the :query-param:`property_ranges` query parameter. - It consist of a list which for each dimension contains a list of three values. + It consists of a list which for each dimension contains a list of three values. The first value indicates the index, in that dimension, of the first value that has been returned. The second value indicates the index of last returned value. The third value is the step size. - It MUST be returned when the `serialization_format` is "regular" or "custom". Otherwise it should not be returned. + It MUST be returned when the `serialization_format` is :val:`"regular"` or :val:`"custom"`. Otherwise, it should not be returned. - :field:`next`: a `JSON API links object `__. If the field is too large to be returned in a single response, this field will contain a URL pointing to the entry, to which this property belongs, which contains the next set of values for this property. - If all the data for this property has been returned its value should be :val:¨`null` + If all the data for this property has been returned, its value should be :val:¨`null` - :field:`more_data_available`: boolean. - :field-val:`false` if all the values in the requested range have been returned and :field-val:`true` if the returned values are incomplete. + :field-val:`false` if all the values in the requested range have been returned, and :field-val:`true` if the returned values are incomplete. Responses @@ -1078,7 +1081,7 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: The third value specifies the step size. Ranges can be specified for multiple properties by separating them with a comma. - The field may include optional space characters, which do no alter the meaning of the expression. + The field may include optional space characters, which do not alter the meaning of the expression. Databases SHOULD return the values falling within this range and the :property:`indexes` or :property:`returned_range` field belonging to the returned values. For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned. For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`. From 916d6f29c500d80300e38be939a009d780d3905e Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 13 Mar 2023 12:15:08 +0100 Subject: [PATCH 18/30] Changed 'n_' to 'n' for ranged metadata properties tio be consistent with nelements. --- optimade.rst | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/optimade.rst b/optimade.rst index e8722399f..3a39cd3b9 100644 --- a/optimade.rst +++ b/optimade.rst @@ -474,7 +474,7 @@ All fields in this section SHOULD be queryable with support for all mandatory fi - `custom`: A separate list with indexes is defined in the field :property:`indexes` to indicate to which index each value belongs. -- :field:`n_indexable_dim`: integer. +- :field:`nindexable_dim`: integer. The number of dimensions that can be indexed. The values themselves can also be lists, so a property may have more dimensions than listed here, but it may not be practical for the server to support indexing at that level. @@ -485,7 +485,7 @@ All fields in this section SHOULD be queryable with support for all mandatory fi Depending on the value of the :field:`serialization_format`, the following fields SHOULD/MUST be present or SHOULD NOT be present in the dictionary. All fields in this section SHOULD be queryable with support for all mandatory filter features. -- :field:`n_values`: integer. +- :field:`nvalues`: integer. The total number of values in the property. This may be larger than the number of values that are returned. This field SHOULD be present when :property:`serialization_format` is not set to :val:`"linear"` else it SHOULD NOT be present. @@ -535,7 +535,7 @@ If the :field:`` contains data, the following properties MUST be The value MUST be present when :property:`serialization_format` is set to :val:`"custom"`. Otherwise, it SHOULD NOT be present. -- :field:`n_returned_values`: integer +- :field:`nreturned_values`: integer The number of values that have been returned. This value SHOULD be present when `serialization_format` is set to :val:`"custom"` or :val:`"regular"`. Otherwise it SHOULD NOT be present. @@ -827,14 +827,14 @@ An example of a full response: ], "cartesian_site_positions_meta": { "range": { - "n_indexable_dim": 3, + "nindexable_dim": 3, "dim_size": [100, 3, 3], "range_ids": ["mdsteps","particles","xyz"], "serialization_format": "regular", "offset_regular": [1, 1, 1], "step_size_regular": [1, 1, 1], - "n_values": 900, - "n_returned_values": 50, + "nvalues": 900, + "nreturned_values": 50, "returned¨_range": [[1,100,2],[1,3,1],[1,3,1]], "more_data_available": true, "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=cartesian_site positions[[101,900,2],[1,3,1],[1,3,1]]" @@ -844,14 +844,14 @@ An example of a full response: "species_at_sites": ["He", "Ne", "Ar"], "species_at_sites_meta": { "range": { - "n_indexable_dim": 1, + "nindexable_dim": 1, "dim_size": [3], "range_ids": ["particles"], "serialization_format": "regular", "offset_regular": [1], "step_size_regular": [1], - "n_returned_values": 3, - "n_values": 3, + "nreturned_values": 3, + "nvalues": 3, "returned_range":[[1,3,1]], "more_data_available": false, "next": null @@ -861,7 +861,7 @@ An example of a full response: "_exmpl_ranged_time": null, "_exmpl_ranged_time_meta":{ "range":{ - "n_indexable_dim": 1, + "nindexable_dim": 1, "dim_size": [100], "range_ids": ["mdsteps"], "serialization_format": "linear", @@ -872,12 +872,12 @@ An example of a full response: "_exmpl_ranged_thermostat": [20, 40, 60], "_exmpl_ranged_thermostat_meta": { "range": { - "n_indexable_dim": 1, + "nindexable_dim": 1, "dim_size": [100], "range_ids": ["mdsteps"], "serialization_format": "custom", - "n_values": 3, - "n_returned_values": 3, + "nvalues": 3, + "nreturned_values": 3, "indexes": [[0], [20], [80]], "more_data_available": false, "next": null From ee3651e9125b7a343dc15fa1eb53902cc1be04de Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 13 Mar 2023 17:18:41 +0100 Subject: [PATCH 19/30] Changed wording of range_id field after suggestion Rickard. --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 3a39cd3b9..ac8698399 100644 --- a/optimade.rst +++ b/optimade.rst @@ -523,8 +523,8 @@ The dictionary MAY include these fields. - :field:`range_ids`: list of strings. A list with an identifier for each dimension of the property. - If two properties have the same range_id for a dimension, it means that the values at an index along this dimension belong to each other. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a trajectory have the same :field:`range_ids` it indicates that the energy at an index x(in the dimension labelled by this range_ids) belongs to the cartesian_site_positions at the same index x. + If dimensions in two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a :field:`range_id` of :val:`frame`, it means that that the energy at an index x(in the dimension labelled by this range_ids) belongs to the cartesian_site_positions at the same index x. SHOULD be a queryable property with support for all mandatory filter features. If the :field:`` contains data, the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. Querying is not relevant for these properties and SHOULD NOT be supported. From 1f794b6ad004c30288204c5bbc1fd2c17353f6fc Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 16 Mar 2023 13:09:17 +0100 Subject: [PATCH 20/30] placed subsequent sentences on seperate lines. --- optimade.rst | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index ac8698399..8bbc5aa2a 100644 --- a/optimade.rst +++ b/optimade.rst @@ -458,7 +458,8 @@ The server can limit the size of the response and offer the values of the proper - **Support**: OPTIONAL support in implementations. - A ranged property can be recognized by the presence of the field :field:`range` in the metadata of the property, i.e. in the field: :field:`_meta`. - - The server may return null or only a part of the values of the property under the field :field:``. In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. + - The server may return null or only a part of the values of the property under the field :field:``. + In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. The dictionary under :field:`_meta.range` MUST include these fields. All fields in this section SHOULD be queryable with support for all mandatory filter features. @@ -487,7 +488,10 @@ All fields in this section SHOULD be queryable with support for all mandatory fi - :field:`nvalues`: integer. - The total number of values in the property. This may be larger than the number of values that are returned. This field SHOULD be present when :property:`serialization_format` is not set to :val:`"linear"` else it SHOULD NOT be present. + The total number of values in the property. + This may be larger than the number of values that are returned. + This field SHOULD be present when :property:`serialization_format` is not set to :val:`"linear"` else it SHOULD NOT be present. + - :field:`step_size_linear`: list of floats. @@ -527,7 +531,8 @@ The dictionary MAY include these fields. For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a :field:`range_id` of :val:`frame`, it means that that the energy at an index x(in the dimension labelled by this range_ids) belongs to the cartesian_site_positions at the same index x. SHOULD be a queryable property with support for all mandatory filter features. -If the :field:`` contains data, the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. Querying is not relevant for these properties and SHOULD NOT be supported. +If the :field:`` contains data, the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. +Querying is not relevant for these properties and SHOULD NOT be supported. - :field:`indexes`: List of lists of integers. @@ -538,16 +543,19 @@ If the :field:`` contains data, the following properties MUST be - :field:`nreturned_values`: integer The number of values that have been returned. - This value SHOULD be present when `serialization_format` is set to :val:`"custom"` or :val:`"regular"`. Otherwise it SHOULD NOT be present. + This value SHOULD be present when `serialization_format` is set to :val:`"custom"` or :val:`"regular"`. + Otherwise it SHOULD NOT be present. - :field:`returned_range`: List of list of integers. - The range belonging to the returned data. It uses the same format as the :query-param:`property_ranges` query parameter. + The range belonging to the returned data. + It uses the same format as the :query-param:`property_ranges` query parameter. It consists of a list which for each dimension contains a list of three values. The first value indicates the index, in that dimension, of the first value that has been returned. The second value indicates the index of last returned value. The third value is the step size. - It MUST be returned when the `serialization_format` is :val:`"regular"` or :val:`"custom"`. Otherwise, it should not be returned. + It MUST be returned when the `serialization_format` is :val:`"regular"` or :val:`"custom"`. + Otherwise, it should not be returned. - :field:`next`: a `JSON API links object `__. From 169a1f439be7130ef54c2f21179703146bd8c15a Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 24 Mar 2023 18:15:22 +0100 Subject: [PATCH 21/30] Processed points discussed with Rickard. --- optimade.rst | 268 ++++++++++++++++++++++++++------------------------- 1 file changed, 138 insertions(+), 130 deletions(-) diff --git a/optimade.rst b/optimade.rst index 8bbc5aa2a..49366d060 100644 --- a/optimade.rst +++ b/optimade.rst @@ -450,122 +450,93 @@ Ranged Properties ----------------- Ranged properties are used for properties that are too large to be returned by default for every entry in a response. -They can also support slicing, so the client can request that only some of the values need to be returned. -The server can limit the size of the response and offer the values of the property in multiple parts. - +The server can limit the size of the response and require that the client performs another query to retrieve the rest of the data. +They can also be used by the server to support slicing, so the client can request that only a subsection of the values need to be returned. - **Requirements/Conventions**: - **Support**: OPTIONAL support in implementations. - A ranged property can be recognized by the presence of the field :field:`range` in the metadata of the property, i.e. in the field: :field:`_meta`. - - The server may return null or only a part of the values of the property under the field :field:``. + - For a ranged property the server can return null or only a part of the values of the property under the field :field:``, so the size of the entries remains limited and many entries can be returned in a single response. In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. The dictionary under :field:`_meta.range` MUST include these fields. -All fields in this section SHOULD be queryable with support for all mandatory filter features. -- :field:`serialization_format`: string. +- :field:`range_ids`: list of strings. + SHOULD be a queryable property with support for all mandatory filter features. + A list with an identifier for each dimension of the property. + If dimensions in two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`, it means that that the energy at an index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. - To improve the compactness of the data, there are several ways to show to which index a value belongs. - The string MUST take one of the following values: +- :field:`indexable_dim`: list of strings. + The list of dimensions for which the client can request a subrange via the property_ranges query parameter. - - `linear`: The value is a linear function of the indexes. - This function is defined by :property:`offset_linear` and :property:`step_size_linear`. - - `regular`: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` indicating the index of the first value. - - `custom`: A separate list with indexes is defined in the field :property:`indexes` to indicate to which index each value belongs. +- :field:`data_range`: list of dictionaries. + This field describes how the values are distributed in the different dimensions. + It consists of a dictionary for each dimension. + This dictionary has the fields: + - :field:`name`: string. + The name of the dimension as given in the :field:`range_ids` field. -- :field:`nindexable_dim`: integer. + - :field:`start`: integer. + The index of the first value of the property in this dimension. + The indexing is 1 based, so the lowest value an index can have is 1. - The number of dimensions that can be indexed. The values themselves can also be lists, so a property may have more dimensions than listed here, but it may not be practical for the server to support indexing at that level. + - :field:`step`: integer. + If the values are regularly spaced in this dimension this value indicates the difference in index between subsequent values. + If there is no regular spacing between the values, the value of this property MUST be :val:`null`. -- :field:`dim_size`: list of integers. + - :field:`stop`: integer. + The index of the last value of this property in this dimension. - The size of the range in each indexable dimension. +- :field:`contains_null`: boolean + This value indicates whether some of the values of the property are :val:`null`. + This may be the case when values are missing or if the data is sparse but the server still wants to present the data as a regular array to enable slicing. + SHOULD be a queryable property with support for all mandatory filter features. + +- :field:`dim_size`: list of integers. -Depending on the value of the :field:`serialization_format`, the following fields SHOULD/MUST be present or SHOULD NOT be present in the dictionary. -All fields in this section SHOULD be queryable with support for all mandatory filter features. + The size of the range for each indexable dimension. + The order is the same as in the :field:`range_ids` field. - :field:`nvalues`: integer. The total number of values in the property. - This may be larger than the number of values that are returned. - This field SHOULD be present when :property:`serialization_format` is not set to :val:`"linear"` else it SHOULD NOT be present. - - -- :field:`step_size_linear`: list of floats. - - If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range. - For example, if the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8. - The value MUST be present when :property:`serialization_format` is set to :val:`"linear"`. - Otherwise, it SHOULD NOT be present. - -- :field:`step_size_regular`: list of integers. - - If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension. - The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`. - Otherwise, it SHOULD NOT be present. - - -Depending on the value of the :property:`serialization_format`, the following fields MAY be present or SHOULD NOT be present in the dictionary. -All fields in this section SHOULD be queryable with support for all mandatory filter features. - -- :field:`offset_linear`: float. - - If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1. - The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value SHOULD NOT be present. - The default value is 0. + SHOULD be a queryable property with support for all mandatory filter features. -- :field:`offset_regular`: list of integers. +- :field:`next`: a `JSON API links object `__. - If :property:`serialization_format` is set to :val:`"regular"` this property gives the indexes of the first value. - The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value SHOULD NOT be present. - The default value is 1 in every dimension. + If there is still more data available for the property, this field will contain a URL pointing to the entry, to which this property belongs, which contains the next set of values for this property. + If all the data for this property has been returned, the value SHOULD be :val:`null` -The dictionary MAY include these fields. +- :field:`more_data_available`: boolean. -- :field:`range_ids`: list of strings. + :field-val:`false` if all the values in the requested range have been returned, and :field-val:`true` if the returned values are incomplete. - A list with an identifier for each dimension of the property. - If dimensions in two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a :field:`range_id` of :val:`frame`, it means that that the energy at an index x(in the dimension labelled by this range_ids) belongs to the cartesian_site_positions at the same index x. - SHOULD be a queryable property with support for all mandatory filter features. -If the :field:`` contains data, the following properties MUST be present or SHOULD NOT be present, depending on the value of the :property:`serialization_format`. +If the :field:`` contains data (i.e. it is not(null or an empty list)), the following additional properties can be present. Querying is not relevant for these properties and SHOULD NOT be supported. -- :field:`indexes`: List of lists of integers. - - If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the field :field:`` belong. - The value MUST be present when :property:`serialization_format` is set to :val:`"custom"`. - Otherwise, it SHOULD NOT be present. - - :field:`nreturned_values`: integer The number of values that have been returned. - This value SHOULD be present when `serialization_format` is set to :val:`"custom"` or :val:`"regular"`. - Otherwise it SHOULD NOT be present. - -- :field:`returned_range`: List of list of integers. + This value SHOULD be present. - The range belonging to the returned data. - It uses the same format as the :query-param:`property_ranges` query parameter. - It consists of a list which for each dimension contains a list of three values. - The first value indicates the index, in that dimension, of the first value that has been returned. - The second value indicates the index of last returned value. - The third value is the step size. - It MUST be returned when the `serialization_format` is :val:`"regular"` or :val:`"custom"`. - Otherwise, it should not be returned. - -- :field:`next`: a `JSON API links object `__. - - If the field is too large to be returned in a single response, this field will contain a URL pointing to the entry, to which this property belongs, which contains the next set of values for this property. - If all the data for this property has been returned, its value should be :val:¨`null` - -- :field:`more_data_available`: boolean. +- :field:`indexes`: list of lists of integers. + If the values are not regularly spaced along the dimensions, this list holds the indexes for each value. + The order of the indexes must match the order in the field :field:`range_ids`. + MUST be present if any of the dimensions in the field :field:`data_ranges.step` has the value null. + i.e. when the values are not regularly distributed over the grid. + Otherwise it SHOULD NOT be present. - :field-val:`false` if all the values in the requested range have been returned, and :field-val:`true` if the returned values are incomplete. +- :field:`returned_range`: list of dictionaries. + The range covering the returned data. + It contains the same fields as the :field:`data_range` field. + However in this case these fields only apply to the returned values and not all the values of the property. + It MUST be present. + For dimensions where the field :field:`data_range.step` is not defined ,the value of the field :field:`returned_range.step` MUST match the stepsize as used in the query parameter :query_param:`property_ranges`. Responses ========= @@ -835,58 +806,97 @@ An example of a full response: ], "cartesian_site_positions_meta": { "range": { - "nindexable_dim": 3, - "dim_size": [100, 3, 3], - "range_ids": ["mdsteps","particles","xyz"], - "serialization_format": "regular", - "offset_regular": [1, 1, 1], - "step_size_regular": [1, 1, 1], - "nvalues": 900, - "nreturned_values": 50, - "returned¨_range": [[1,100,2],[1,3,1],[1,3,1]], + "range_ids": ["frames","particles","xyz"], + "indexable_dim": ["frames","particles","xyz"], + "data_range": [{ + "name": "frames", + "start": 1, + "step": 1, + "stop": 200, + },{ + "name": "particles", + "start": 1, + "step": 1, + "stop": 3, + },{ + "name": "xyz", + "start": 1, + "step": 1, + "stop": 3, + }], + "dim_size": [200, 3, 3], + "nvalues": 1800, + "nreturned_values": 450, + "returned_range": [{ + "name": "frames", + "start": 1, + "step": 2, + "stop": 100, + },{ + "name": "particles", + "start": 1, + "step": 1, + "stop": 3, + },{ + "name": "xyz", + "start": 1, + "step": 1, + "stop": 3, + }], + "contains_null": false, "more_data_available": true, - "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=cartesian_site positions[[101,900,2],[1,3,1],[1,3,1]]" + "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=mdstep[101,200,2],particles[1,3,1],xyz[1,3,1]]" }, // ... }, "species_at_sites": ["He", "Ne", "Ar"], "species_at_sites_meta": { "range": { - "nindexable_dim": 1, + "indexable_dim": ["particles"], "dim_size": [3], "range_ids": ["particles"], - "serialization_format": "regular", - "offset_regular": [1], - "step_size_regular": [1], + "data_range": [{ + "name": "particles", + "start": 1, + "step": 1, + "stop": 3, + }], "nreturned_values": 3, "nvalues": 3, - "returned_range":[[1,3,1]], + "returned_range":[{ + "name": "particles", + "start": 1, + "step": 1, + "stop": 3, + }], + "contains_null": false, "more_data_available": false, "next": null }, // ... }, - "_exmpl_ranged_time": null, - "_exmpl_ranged_time_meta":{ - "range":{ - "nindexable_dim": 1, - "dim_size": [100], - "range_ids": ["mdsteps"], - "serialization_format": "linear", - "step_size_linear": 0.2 - }, - // ... - }, "_exmpl_ranged_thermostat": [20, 40, 60], "_exmpl_ranged_thermostat_meta": { "range": { - "nindexable_dim": 1, - "dim_size": [100], - "range_ids": ["mdsteps"], - "serialization_format": "custom", + "indexable_dim":["frames"], + "dim_size": [200], + "data_range": [{ + "name": "particles", + "start": 1, + "step": null, + "stop": 80, + }], + "range_ids": ["frames"], "nvalues": 3, "nreturned_values": 3, - "indexes": [[0], [20], [80]], + "indexes": [[1], [20], [80]], + "contains_null": false, + "returned_range":[{ + "name": "particles", + "start": 1, + "step": 1, + "stop": 80, + }], "more_data_available": false, "next": null }, @@ -1078,30 +1088,28 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` -- **property\_ranges**: specifies which data ranges should be returned for ranged properties. +- **property\_ranges**: specifies which ranges should be used when returning ranged properties. In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - It consists of a property name directly followed by the range that should be returned. - A range is a list containing a list for each dimension. - A list consists of a pair of square brackets ("[", ASCII 91(0x5B)) and ("]", ASCII 93(0x5D)) enclosing a number of values separated by commas (",", ASCII 91(0x5B)) - Each dimension's list has three integer values. - The first value of the range specifies the first index in that dimension for which values should be returned. - The second value specifies the last index for which values should be returned. - The third value specifies the step size. - - Ranges can be specified for multiple properties by separating them with a comma. + It consists of the name of a dimension directly followed by a range. + A range consists of a pair of brackets ("(", ASCII 40(0x28)) and (")", ASCII 41(0x29)) enclosing three integers, which are separated by commas (",", ASCII 91(0x5B)) + The first integer specifies the first index in that dimension for which values should be returned. + The second integer specifies the last index for which values should be returned. + The third integer specifies the step size in that dimension. + Ranges can be specified for multiple dimensions by separating them with a comma. The field may include optional space characters, which do not alter the meaning of the expression. - Databases SHOULD return the values falling within this range and the :property:`indexes` or :property:`returned_range` field belonging to the returned values. - For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned. - For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`. - The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`[10,20,1]` the last value returned belongs to index 20. + Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the Database MAY return more data than was specified in the range. + The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`(10,20,1)` the last value returned belongs to index 20. + If a dimension is not specified it is assumed that the whole range in that dimension is requested. + If a value is not present at a set of the indexes, no value SHOULD be returned. + However, when a value is explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned. + Example: - A database has a :entry:`structure` entry with id: :val:`id_12345` and a ranged property :property:`test_field` with the two-dimensional data values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]]`. - A client makes a request :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=test_field[[1, 3, 2], [2, 3, 1]]`. + A database has a :entry:`structure` entry with id: :val:`id_12345` and a ranged property :property:`test_field` with the two-dimensional data values :val:`[[9.64, 7.52, 0.69], [4.82, 8.35, 3.26], [4.82, 2.78, 7.87], [5.49, 3.48, 1.65]]`. + In addition the field :field:`range_ids` has the value :val:"["frames", "xyz"] + A client can then make the request :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=frames(1, 4, 2),xyz(2, 3, 1)&response_fields=test_field`. The response is then a single entry response for structure `12345` where the `test_field` property is included with the values :val:`[[7.52, 0.69], [2.78, 7.87]]`. - Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=test_field[[1, 3, 2], [2, 3, 1]], other_field[[1,100,1]]`. - Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". From f513596718f1cfbad383a77913db607a56b3bb2e Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 28 Mar 2023 18:47:38 +0200 Subject: [PATCH 22/30] Added per entry next field + small corrections. --- optimade.rst | 42 ++++++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/optimade.rst b/optimade.rst index 49366d060..ec3dea8f4 100644 --- a/optimade.rst +++ b/optimade.rst @@ -451,25 +451,24 @@ Ranged Properties Ranged properties are used for properties that are too large to be returned by default for every entry in a response. The server can limit the size of the response and require that the client performs another query to retrieve the rest of the data. -They can also be used by the server to support slicing, so the client can request that only a subsection of the values need to be returned. +They can also be used by the server to support slicing, so the client can request that only a subsection of the values needs to be returned. - **Requirements/Conventions**: - **Support**: OPTIONAL support in implementations. - A ranged property can be recognized by the presence of the field :field:`range` in the metadata of the property, i.e. in the field: :field:`_meta`. - - For a ranged property the server can return null or only a part of the values of the property under the field :field:``, so the size of the entries remains limited and many entries can be returned in a single response. - In that case a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. + - For a ranged property, the server MAY return :val:`null` or only a part of the values of the property under the field :field:``, so the size of the entries remains limited, and many entries can be returned in a single response. + In that case, a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. The dictionary under :field:`_meta.range` MUST include these fields. - :field:`range_ids`: list of strings. - SHOULD be a queryable property with support for all mandatory filter features. A list with an identifier for each dimension of the property. If dimensions in two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`, it means that that the energy at an index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + For example, both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. - :field:`indexable_dim`: list of strings. - The list of dimensions for which the client can request a subrange via the property_ranges query parameter. + The list of dimensions for which the client can request a subrange via the :query-param:`property_ranges` query parameter. - :field:`data_range`: list of dictionaries. This field describes how the values are distributed in the different dimensions. @@ -484,7 +483,7 @@ The dictionary under :field:`_meta.range` MUST include these fiel The indexing is 1 based, so the lowest value an index can have is 1. - :field:`step`: integer. - If the values are regularly spaced in this dimension this value indicates the difference in index between subsequent values. + If the values are regularly spaced in this dimension, this value indicates the difference in index between subsequent values. If there is no regular spacing between the values, the value of this property MUST be :val:`null`. - :field:`stop`: integer. @@ -492,7 +491,7 @@ The dictionary under :field:`_meta.range` MUST include these fiel - :field:`contains_null`: boolean This value indicates whether some of the values of the property are :val:`null`. - This may be the case when values are missing or if the data is sparse but the server still wants to present the data as a regular array to enable slicing. + This may be the case when values are missing or if the data is sparse, but the server still wants to present the data as a regular array to enable slicing. SHOULD be a queryable property with support for all mandatory filter features. - :field:`dim_size`: list of integers. @@ -515,7 +514,7 @@ The dictionary under :field:`_meta.range` MUST include these fiel :field-val:`false` if all the values in the requested range have been returned, and :field-val:`true` if the returned values are incomplete. -If the :field:`` contains data (i.e. it is not(null or an empty list)), the following additional properties can be present. +If the :field:`` contains data (i.e. it is not (null or an empty list)), the following additional properties can be present. Querying is not relevant for these properties and SHOULD NOT be supported. - :field:`nreturned_values`: integer @@ -528,15 +527,27 @@ Querying is not relevant for these properties and SHOULD NOT be supported. The order of the indexes must match the order in the field :field:`range_ids`. MUST be present if any of the dimensions in the field :field:`data_ranges.step` has the value null. i.e. when the values are not regularly distributed over the grid. - Otherwise it SHOULD NOT be present. + Otherwise, it SHOULD NOT be present. - :field:`returned_range`: list of dictionaries. The range covering the returned data. It contains the same fields as the :field:`data_range` field. - However in this case these fields only apply to the returned values and not all the values of the property. - It MUST be present. - For dimensions where the field :field:`data_range.step` is not defined ,the value of the field :field:`returned_range.step` MUST match the stepsize as used in the query parameter :query_param:`property_ranges`. + In this case these fields, however, only apply to the returned values and not all the values of the property. + This field MUST be present. + For dimensions where the field :field:`data_range.step` is not defined, the value of the field :field:`returned_range.step` MUST match the stepsize as used in the query parameter :query_param:`property_ranges`. + +In addition to these fields in the metadata, entries which support accessing data via the :query-param:`property\_ranges` query parameter SHOULD support per entry :field:`next` and :field:`more_data_availble` field to enable returning all the + +- :field:`next`: a `JSON API links object `__. + If data is requested for multiple properties at the same time, but the total amount of data is too large to be returned in one response, this field contains a link from which the remainder of the data can be obtained. + Responses supplied via this next link MUST contain all the values for all the requested properties that lie within the returned range. The server May choose to return a range that is smaller than the requested range. In that case another next link SHOULD be provided. + If all the data for this property has been returned, the value SHOULD be :val:`null` + +- :field:`more_data_available`: boolean. + + :field-val:`false` if all the values in the requested range have been returned, and :field-val:`true` if the returned values are incomplete. + Responses ========= @@ -1096,17 +1107,16 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: The second integer specifies the last index for which values should be returned. The third integer specifies the step size in that dimension. Ranges can be specified for multiple dimensions by separating them with a comma. - The field may include optional space characters, which do not alter the meaning of the expression. Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the Database MAY return more data than was specified in the range. The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`(10,20,1)` the last value returned belongs to index 20. - If a dimension is not specified it is assumed that the whole range in that dimension is requested. + If a dimension is not specified, it is assumed that the whole range in that dimension is requested. If a value is not present at a set of the indexes, no value SHOULD be returned. However, when a value is explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned. Example: A database has a :entry:`structure` entry with id: :val:`id_12345` and a ranged property :property:`test_field` with the two-dimensional data values :val:`[[9.64, 7.52, 0.69], [4.82, 8.35, 3.26], [4.82, 2.78, 7.87], [5.49, 3.48, 1.65]]`. - In addition the field :field:`range_ids` has the value :val:"["frames", "xyz"] + In addition, the field :field:`range_ids` has the value :val:"["frames", "xyz"] A client can then make the request :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=frames(1, 4, 2),xyz(2, 3, 1)&response_fields=test_field`. The response is then a single entry response for structure `12345` where the `test_field` property is included with the values :val:`[[7.52, 0.69], [2.78, 7.87]]`. From 65b8ad153db963938b85b9f88649f9c8b5290caa Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 2 May 2023 12:22:20 +0200 Subject: [PATCH 23/30] Apply suggestions from code review Co-authored-by: Rickard Armiento --- optimade.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index ec3dea8f4..e05158417 100644 --- a/optimade.rst +++ b/optimade.rst @@ -21,7 +21,6 @@ OPTIMADE API specification v1.2.0~develop entry : names of type of resources, served via OPTIMADE, pertaining to data in a database. property : data item that belongs to an entry. - ranged_property : A property that can be returned in pieces and that supports slicing. val : value examples that properties can be. :val: is ONLY used when referencing values of actual properties, i.e., information that belongs to the database. type : data type of values. @@ -460,7 +459,7 @@ They can also be used by the server to support slicing, so the client can reques - For a ranged property, the server MAY return :val:`null` or only a part of the values of the property under the field :field:``, so the size of the entries remains limited, and many entries can be returned in a single response. In that case, a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. -The dictionary under :field:`_meta.range` MUST include these fields. +The metadata field of the ranged property, :field:`_meta.range`, MUST include these fields: - :field:`range_ids`: list of strings. A list with an identifier for each dimension of the property. @@ -514,7 +513,7 @@ The dictionary under :field:`_meta.range` MUST include these fiel :field-val:`false` if all the values in the requested range have been returned, and :field-val:`true` if the returned values are incomplete. -If the :field:`` contains data (i.e. it is not (null or an empty list)), the following additional properties can be present. +If the :field:`` contains data, i.e., it is neither :val:`null`, nor the empty list, the following additional properties can be present: Querying is not relevant for these properties and SHOULD NOT be supported. - :field:`nreturned_values`: integer From 94db38c5e1a052377e2ba800ae608993598706c4 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 2 May 2023 14:34:27 +0200 Subject: [PATCH 24/30] Adjusted the description next fields for ranged properties. --- optimade.rst | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/optimade.rst b/optimade.rst index e05158417..e96d20be4 100644 --- a/optimade.rst +++ b/optimade.rst @@ -67,8 +67,6 @@ OPTIMADE API specification v1.2.0~develop .. role:: property(literal) -.. role:: ranged-property(literal) - .. role:: val(literal) .. role:: type(literal) @@ -503,9 +501,14 @@ The metadata field of the ranged property, :field:`_meta.range`, The total number of values in the property. SHOULD be a queryable property with support for all mandatory filter features. -- :field:`next`: a `JSON API links object `__. +- :field:`next`: `JSON API links object `_ + + If there is still more data available for the property, this field MUST contain a URL from which the next set of values for this property can be obtained. + The `JSON API links object `__, containing the URL, is either a string, or a links object, which can contain the following fields: + + - **href**: a string containing the URL. + - **meta**: a meta object containing non-standard meta-information about the next link. - If there is still more data available for the property, this field will contain a URL pointing to the entry, to which this property belongs, which contains the next set of values for this property. If all the data for this property has been returned, the value SHOULD be :val:`null` - :field:`more_data_available`: boolean. @@ -536,13 +539,19 @@ Querying is not relevant for these properties and SHOULD NOT be supported. This field MUST be present. For dimensions where the field :field:`data_range.step` is not defined, the value of the field :field:`returned_range.step` MUST match the stepsize as used in the query parameter :query_param:`property_ranges`. -In addition to these fields in the metadata, entries which support accessing data via the :query-param:`property\_ranges` query parameter SHOULD support per entry :field:`next` and :field:`more_data_availble` field to enable returning all the +In addition to these fields in the metadata, entries which support accessing data via the :query-param:`property\_ranges` query parameter SHOULD support per entry :field:`next` and :field:`more_data_available` field to enable returning the remainder of the data for all the properties for the rest of the range. + +- :field:`next`: `JSON API links object `__. -- :field:`next`: a `JSON API links object `__. If data is requested for multiple properties at the same time, but the total amount of data is too large to be returned in one response, this field contains a link from which the remainder of the data can be obtained. Responses supplied via this next link MUST contain all the values for all the requested properties that lie within the returned range. The server May choose to return a range that is smaller than the requested range. In that case another next link SHOULD be provided. If all the data for this property has been returned, the value SHOULD be :val:`null` + The `JSON API links object `__, containing the URL, is either a string, or a links object, which can contain the following fields: + + - **href**: a string containing the URL. + - **meta**: a meta object containing non-standard meta-information about the next link. + - :field:`more_data_available`: boolean. :field-val:`false` if all the values in the requested range have been returned, and :field-val:`true` if the returned values are incomplete. @@ -855,7 +864,7 @@ An example of a full response: }], "contains_null": false, "more_data_available": true, - "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=mdstep[101,200,2],particles[1,3,1],xyz[1,3,1]]" + "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=mdstep(101,200,2),particles(1,3,1),xyz(1,3,1)" }, // ... }, From 033ea118bedc2adcdc61dfbaff242488bfc9b14f Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 25 May 2023 14:53:06 +0200 Subject: [PATCH 25/30] Corrected range name for _exmpl_ranged_thermostat. --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index e96d20be4..6193efb84 100644 --- a/optimade.rst +++ b/optimade.rst @@ -900,7 +900,7 @@ An example of a full response: "indexable_dim":["frames"], "dim_size": [200], "data_range": [{ - "name": "particles", + "name": "frames", "start": 1, "step": null, "stop": 80, @@ -911,9 +911,9 @@ An example of a full response: "indexes": [[1], [20], [80]], "contains_null": false, "returned_range":[{ - "name": "particles", + "name": "frames", "start": 1, - "step": 1, + "step": null, "stop": 80, }], "more_data_available": false, From 3762d30092726ece1729ffc0c373b74b4880456d Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 30 May 2023 14:24:56 +0200 Subject: [PATCH 26/30] Added that querying on properties in the range dictionary is optional. --- optimade.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index 6193efb84..932828541 100644 --- a/optimade.rst +++ b/optimade.rst @@ -456,16 +456,18 @@ They can also be used by the server to support slicing, so the client can reques - A ranged property can be recognized by the presence of the field :field:`range` in the metadata of the property, i.e. in the field: :field:`_meta`. - For a ranged property, the server MAY return :val:`null` or only a part of the values of the property under the field :field:``, so the size of the entries remains limited, and many entries can be returned in a single response. In that case, a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. + - Support for queries on the fields under :field:`range` is OPTIONAL. The metadata field of the ranged property, :field:`_meta.range`, MUST include these fields: - :field:`range_ids`: list of strings. A list with an identifier for each dimension of the property. - If dimensions in two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. - :field:`indexable_dim`: list of strings. - The list of dimensions for which the client can request a subrange via the :query-param:`property_ranges` query parameter. + The list of range_ids of the dimensions for which the client can request a subrange via the :query-param:`property_ranges` query parameter. - :field:`data_range`: list of dictionaries. This field describes how the values are distributed in the different dimensions. From 83325679dd21f11e300fe764cffced9da2aba026 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 31 May 2023 15:54:57 +0200 Subject: [PATCH 27/30] Specifically mention that support for queries directly on the values of ranged properties is Optional. --- optimade.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 932828541..735fee7d0 100644 --- a/optimade.rst +++ b/optimade.rst @@ -457,6 +457,7 @@ They can also be used by the server to support slicing, so the client can reques - For a ranged property, the server MAY return :val:`null` or only a part of the values of the property under the field :field:``, so the size of the entries remains limited, and many entries can be returned in a single response. In that case, a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. - Support for queries on the fields under :field:`range` is OPTIONAL. + - As ranged properties can have many values, support for queries on theses values is OPTIONAL. The metadata field of the ranged property, :field:`_meta.range`, MUST include these fields: @@ -467,7 +468,7 @@ The metadata field of the ranged property, :field:`_meta.range`, This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. - :field:`indexable_dim`: list of strings. - The list of range_ids of the dimensions for which the client can request a subrange via the :query-param:`property_ranges` query parameter. + The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange via the :query-param:`property_ranges` query parameter. - :field:`data_range`: list of dictionaries. This field describes how the values are distributed in the different dimensions. From a87b30134a64674c27cc20864f2c66fd208c0e63 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 2 Jun 2023 14:52:51 +0200 Subject: [PATCH 28/30] Updated example to latest version metadata proposal and added more explanation for property_ranges query parameter. --- optimade.rst | 41 ++++++++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/optimade.rst b/optimade.rst index 735fee7d0..8affa3bcf 100644 --- a/optimade.rst +++ b/optimade.rst @@ -492,7 +492,6 @@ The metadata field of the ranged property, :field:`_meta.range`, - :field:`contains_null`: boolean This value indicates whether some of the values of the property are :val:`null`. This may be the case when values are missing or if the data is sparse, but the server still wants to present the data as a regular array to enable slicing. - SHOULD be a queryable property with support for all mandatory filter features. - :field:`dim_size`: list of integers. @@ -820,13 +819,19 @@ An example of a full response: .. code:: jsonc - { + { + "attributes":{ "cartesian_site_positions": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]], [[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]], [[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]] // ... ], - "cartesian_site_positions_meta": { + "species_at_sites": ["He", "Ne", "Ar"], + "_exmpl_ranged_thermostat": [20, 40, 60], + // ... + }, + "meta":{ + "cartesian_site_positions": { "range": { "range_ids": ["frames","particles","xyz"], "indexable_dim": ["frames","particles","xyz"], @@ -848,7 +853,7 @@ An example of a full response: }], "dim_size": [200, 3, 3], "nvalues": 1800, - "nreturned_values": 450, + "nreturned_values": 900, "returned_range": [{ "name": "frames", "start": 1, @@ -867,12 +872,10 @@ An example of a full response: }], "contains_null": false, "more_data_available": true, - "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=mdstep(101,200,2),particles(1,3,1),xyz(1,3,1)" - }, - // ... + "next": "https://example.com/optimade/v1/structures/id123456?response_fields=cartesian_site_positions&property_ranges=frames(101,200,2),particles(1,3,1),xyz(1,3,1)" + } }, - "species_at_sites": ["He", "Ne", "Ar"], - "species_at_sites_meta": { + "species_at_sites": { "range": { "indexable_dim": ["particles"], "dim_size": [3], @@ -894,11 +897,9 @@ An example of a full response: "contains_null": false, "more_data_available": false, "next": null - }, - // ... + } }, - "_exmpl_ranged_thermostat": [20, 40, 60], - "_exmpl_ranged_thermostat_meta": { + "_exmpl_ranged_thermostat": { "range": { "indexable_dim":["frames"], "dim_size": [200], @@ -921,10 +922,11 @@ An example of a full response: }], "more_data_available": false, "next": null - }, - // ... + } } - } + }, + // ... +} HTTP Response Status Codes @@ -1118,11 +1120,16 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: The second integer specifies the last index for which values should be returned. The third integer specifies the step size in that dimension. Ranges can be specified for multiple dimensions by separating them with a comma. - Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the Database MAY return more data than was specified in the range. + Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for the range :val:`(10,20,1)` the last value returned belongs to index 20. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. If a value is not present at a set of the indexes, no value SHOULD be returned. However, when a value is explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned. + Incase the requested property_range covers more data than the server wants to return the server may choose to return only a part of the data. + For each combination of indexes for which data is returned all the values for all requested properties however need to be returned. + If the server does not return all the requested data, a link MUST be provided in the :field:`next` field, that applies to an entry as a whole, from which the remainder of the data can be retrieved. + + Example: From 7e9b4f4f45dc6ac5079b46151b94f28a1ed66732 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 2 Jun 2023 16:35:42 +0200 Subject: [PATCH 29/30] Improved explanation returned_range field. --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 8affa3bcf..173d6aa20 100644 --- a/optimade.rst +++ b/optimade.rst @@ -536,12 +536,12 @@ Querying is not relevant for these properties and SHOULD NOT be supported. - :field:`returned_range`: list of dictionaries. The range covering the returned data. - It contains the same fields as the :field:`data_range` field. + The dictionaries contain the same fields as those of the :field:`data_range` field. In this case these fields, however, only apply to the returned values and not all the values of the property. This field MUST be present. For dimensions where the field :field:`data_range.step` is not defined, the value of the field :field:`returned_range.step` MUST match the stepsize as used in the query parameter :query_param:`property_ranges`. -In addition to these fields in the metadata, entries which support accessing data via the :query-param:`property\_ranges` query parameter SHOULD support per entry :field:`next` and :field:`more_data_available` field to enable returning the remainder of the data for all the properties for the rest of the range. +In addition to these fields in the metadata, entries which support accessing data via the :query-param:`property\_ranges` query parameter SHOULD support per entry :field:`next` and :field:`more_data_available` fields to enable returning the remainder of the data for all properties for the rest of the range. - :field:`next`: `JSON API links object `__. From 4b298c54026ff6476e66e7edd3fd908e9af7671b Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 2 Jun 2023 18:51:38 +0200 Subject: [PATCH 30/30] Small corrections. --- optimade.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/optimade.rst b/optimade.rst index 173d6aa20..388f65f6c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -447,19 +447,20 @@ Ranged Properties ----------------- Ranged properties are used for properties that are too large to be returned by default for every entry in a response. -The server can limit the size of the response and require that the client performs another query to retrieve the rest of the data. -They can also be used by the server to support slicing, so the client can request that only a subsection of the values needs to be returned. +The server can therefore choose to limit the size of the response by not returning (all) the values. +The client is then has to perform another query to retrieve the rest of the data. +Ranged Properties can also be used by the server to support slicing, so the client can request that only a subsection of the values needs to be returned. - **Requirements/Conventions**: - **Support**: OPTIONAL support in implementations. - - A ranged property can be recognized by the presence of the field :field:`range` in the metadata of the property, i.e. in the field: :field:`_meta`. + - A ranged property can be recognized by the presence of the field :field:`range` in the metadata of the property, i.e. in the field: :field:`` under the per entry :field: ̀meta` field. - For a ranged property, the server MAY return :val:`null` or only a part of the values of the property under the field :field:``, so the size of the entries remains limited, and many entries can be returned in a single response. - In that case, a links object MUST be provided in the field :field:`_meta.range.next` from which the next part of the property is returned. + In that case, a links object MUST be provided in the field :field:`meta..range.next` from which the next part of the property is returned. - Support for queries on the fields under :field:`range` is OPTIONAL. - As ranged properties can have many values, support for queries on theses values is OPTIONAL. -The metadata field of the ranged property, :field:`_meta.range`, MUST include these fields: +The metadata field of the ranged property, :field:`meta..range`, MUST include these fields: - :field:`range_ids`: list of strings. A list with an identifier for each dimension of the property. @@ -518,7 +519,7 @@ The metadata field of the ranged property, :field:`_meta.range`, :field-val:`false` if all the values in the requested range have been returned, and :field-val:`true` if the returned values are incomplete. -If the :field:`` contains data, i.e., it is neither :val:`null`, nor the empty list, the following additional properties can be present: +If the :field:`` contains data, i.e., it is neither :val:`null`, nor an empty list, the following additional properties can be present: Querying is not relevant for these properties and SHOULD NOT be supported. - :field:`nreturned_values`: integer @@ -527,6 +528,7 @@ Querying is not relevant for these properties and SHOULD NOT be supported. This value SHOULD be present. - :field:`indexes`: list of lists of integers. + If the values are not regularly spaced along the dimensions, this list holds the indexes for each value. The order of the indexes must match the order in the field :field:`range_ids`. MUST be present if any of the dimensions in the field :field:`data_ranges.step` has the value null. @@ -538,7 +540,7 @@ Querying is not relevant for these properties and SHOULD NOT be supported. The range covering the returned data. The dictionaries contain the same fields as those of the :field:`data_range` field. In this case these fields, however, only apply to the returned values and not all the values of the property. - This field MUST be present. + This field MUST be present when values are returned. For dimensions where the field :field:`data_range.step` is not defined, the value of the field :field:`returned_range.step` MUST match the stepsize as used in the query parameter :query_param:`property_ranges`. In addition to these fields in the metadata, entries which support accessing data via the :query-param:`property\_ranges` query parameter SHOULD support per entry :field:`next` and :field:`more_data_available` fields to enable returning the remainder of the data for all properties for the rest of the range. @@ -546,8 +548,8 @@ In addition to these fields in the metadata, entries which support accessing dat - :field:`next`: `JSON API links object `__. If data is requested for multiple properties at the same time, but the total amount of data is too large to be returned in one response, this field contains a link from which the remainder of the data can be obtained. - Responses supplied via this next link MUST contain all the values for all the requested properties that lie within the returned range. The server May choose to return a range that is smaller than the requested range. In that case another next link SHOULD be provided. - If all the data for this property has been returned, the value SHOULD be :val:`null` + Responses supplied via this next link MUST contain all the values for all the requested properties that lie within the requested range and have not yet been returned. The server MAY again choose to return only a part of the values. In that case another next link SHOULD be provided for the remaining values. + If all the data for this entryy has been returned, the value SHOULD be :val:`null` The `JSON API links object `__, containing the URL, is either a string, or a links object, which can contain the following fields: @@ -1130,7 +1132,6 @@ Standard OPTIONAL URL query parameters not in the JSON API specification: If the server does not return all the requested data, a link MUST be provided in the :field:`next` field, that applies to an entry as a whole, from which the remainder of the data can be retrieved. - Example: A database has a :entry:`structure` entry with id: :val:`id_12345` and a ranged property :property:`test_field` with the two-dimensional data values :val:`[[9.64, 7.52, 0.69], [4.82, 8.35, 3.26], [4.82, 2.78, 7.87], [5.49, 3.48, 1.65]]`.