From 4170937059323b140438ab10a45f6de44b09e1dc Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 6 Apr 2023 19:02:51 +0200 Subject: [PATCH 01/39] First draft metadata field. --- optimade.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/optimade.rst b/optimade.rst index 2947c3fcf..26e312d38 100644 --- a/optimade.rst +++ b/optimade.rst @@ -442,6 +442,28 @@ For example, the following query can be sent to API implementations `exmpl1` and :filter:`filter=_exmpl1_band_gap<2.0 OR _exmpl2_band_gap<2.5` +Per property metadata fields +---------------------------- + +Implementations are allowed to specify an OPTIONAL field containing per entry metadata for a property. +The name of the metadata field consists of the name of the property for which it contains the metadata, suffixed with "_meta". +For example, when the field is :property:`cartesian_site_positions` the metadata field would be :field:`cartesian_site_positions_meta`. + +This metadata field consists of a dictionary which MAY contain database specific fields. +If an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. + +Example: + + .. code:: jsonc + + { + "element_ratios":[0.3333333333333333, 0.2222222222222222, 0.4444444444444444], + "element_ratios_meta": { + "_exmpl_95_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44356,0.44458], + //... + } + + Responses ========= From 92c4d923e8279e1d4a57bfb78301334231ecefbe Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 11 Apr 2023 13:26:20 +0200 Subject: [PATCH 02/39] updated example so the numbers make sense. --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 26e312d38..6bfb10765 100644 --- a/optimade.rst +++ b/optimade.rst @@ -457,9 +457,9 @@ Example: .. code:: jsonc { - "element_ratios":[0.3333333333333333, 0.2222222222222222, 0.4444444444444444], + "element_ratios":[0.33336, 0.22229, 0.44425], "element_ratios_meta": { - "_exmpl_95_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44356,0.44458], + "_exmpl_95_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458], //... } From 6fae33d76807e28166e0e3ce7722b02fc0f1b4b6 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 11 Apr 2023 14:27:45 +0200 Subject: [PATCH 03/39] Added that metadata fields can have their own metadata. --- optimade.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 6bfb10765..c04205497 100644 --- a/optimade.rst +++ b/optimade.rst @@ -450,8 +450,10 @@ The name of the metadata field consists of the name of the property for which it For example, when the field is :property:`cartesian_site_positions` the metadata field would be :field:`cartesian_site_positions_meta`. This metadata field consists of a dictionary which MAY contain database specific fields. +Fields in this metadata MAY also have a metadata field. If an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. + Example: .. code:: jsonc @@ -459,7 +461,10 @@ Example: { "element_ratios":[0.33336, 0.22229, 0.44425], "element_ratios_meta": { - "_exmpl_95_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458], + "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], + "_exmpl_confidence_interval_meta":{ + "_exmpl_confidence_level": 0.95 + } //... } From d343a86b31f494ff558851a5a8bd823977fd1b11 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 18 Apr 2023 13:43:45 +0200 Subject: [PATCH 04/39] Add specifically that meta suffix must not be used other than for metadata fields. --- optimade.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index c04205497..0e20d0f11 100644 --- a/optimade.rst +++ b/optimade.rst @@ -379,13 +379,20 @@ Visiting this URL in a web browser gives a human-readable description of how to API implementations SHOULD NOT make up and use new prefixes without first getting them registered in the official list. +Database-specific fields +~~~~~~~~~~~~~~~~~~~~~~~~ + +Database-specific fields MUST be prefixed with (one of) the prefix(es) registered by the database provider enclosed by underscores("\_", ASCII 95 dec (0x5F)). + **Examples**: -- A database-provider-specific prefix: ``exmpl``. Used as a field name in a response: :field:`_exmpl_custom_field`. +- A database-provider-specific prefix: ``exmpl``. Used in a field name in a response: :field:`_exmpl_custom_field`. The initial underscore indicates an identifier that is under a separate namespace under the ownership of that organization. Identifiers prefixed with underscores will not be used for standardized names. +The suffix _meta is reserved for properties that contain the metadata, as described in the section `Per property metadata fields`_ and MUST not be used for fields that do not follow the guidelines described there. + URL Encoding ------------ From 54b632c810cfd6d3f0bc09e07abe94afdee7c32f Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 18 Apr 2023 14:00:23 +0200 Subject: [PATCH 05/39] Moved database specific fields to a seperate section and renamed them to Database-Provider-Specific fields. --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index 0e20d0f11..6780d89b4 100644 --- a/optimade.rst +++ b/optimade.rst @@ -379,10 +379,10 @@ Visiting this URL in a web browser gives a human-readable description of how to API implementations SHOULD NOT make up and use new prefixes without first getting them registered in the official list. -Database-specific fields -~~~~~~~~~~~~~~~~~~~~~~~~ +Database-Provider-Specific fields +--------------------------------- -Database-specific fields MUST be prefixed with (one of) the prefix(es) registered by the database provider enclosed by underscores("\_", ASCII 95 dec (0x5F)). +Database-Provider-Specific fields MUST be prefixed with (one of) the prefix(es) registered by the database provider enclosed by underscores("\_", ASCII 95 dec (0x5F)). **Examples**: From 5d186d93dd17c1ab0a16bf65ba63a22f69941752 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 18 Apr 2023 14:48:23 +0200 Subject: [PATCH 06/39] Apply suggestions from code review Code suggestions from Vaitkus. Co-authored-by: Antanas Vaitkus --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 6780d89b4..bfbbbfaa0 100644 --- a/optimade.rst +++ b/optimade.rst @@ -382,7 +382,7 @@ API implementations SHOULD NOT make up and use new prefixes without first gettin Database-Provider-Specific fields --------------------------------- -Database-Provider-Specific fields MUST be prefixed with (one of) the prefix(es) registered by the database provider enclosed by underscores("\_", ASCII 95 dec (0x5F)). +Database-Provider-Specific fields MUST be prefixed with one of the prefixes registered by the database provider enclosed by underscores ("\_", ASCII 95 dec (0x5F)). **Examples**: @@ -391,7 +391,7 @@ Database-Provider-Specific fields MUST be prefixed with (one of) the prefix(es) The initial underscore indicates an identifier that is under a separate namespace under the ownership of that organization. Identifiers prefixed with underscores will not be used for standardized names. -The suffix _meta is reserved for properties that contain the metadata, as described in the section `Per property metadata fields`_ and MUST not be used for fields that do not follow the guidelines described there. +The suffix ``_meta`` is reserved for properties that contain the metadata, as described in the section `Per property metadata fields`_ and MUST not be used for fields that do not follow the guidelines described there. URL Encoding ------------ From cb63bf2c005a166267d416c67a57880424485b9b Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 25 Apr 2023 18:01:07 +0200 Subject: [PATCH 07/39] Added 'metadata_for' field. --- optimade.rst | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 6780d89b4..c58a3fe7f 100644 --- a/optimade.rst +++ b/optimade.rst @@ -460,17 +460,25 @@ This metadata field consists of a dictionary which MAY contain database specific Fields in this metadata MAY also have a metadata field. If an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. +The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_. +If two metadata fields have a sub field, that is present in both meta data fields, both these sub fields SHOULD have the same $id. -Example: +It SHOULD contain the field : + +- **metadata_for**: which contains a string with the name of the property for which this meta data property contains the meta data, + +Example of a returned metadata field: .. code:: jsonc { "element_ratios":[0.33336, 0.22229, 0.44425], "element_ratios_meta": { + "metadata_for": "element_ratios", "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], "_exmpl_confidence_interval_meta":{ - "_exmpl_confidence_level": 0.95 + "_exmpl_confidence_level": 0.95, + "metadata_for": "_exmpl_confidence_interval", } //... } From d0a4e6544d948ea693c7d1d6c9a91d95305cac84 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 25 Apr 2023 18:39:27 +0200 Subject: [PATCH 08/39] Revert "Moved database specific fields to a seperate section and renamed them to Database-Provider-Specific fields." This reverts commit 54b632c810cfd6d3f0bc09e07abe94afdee7c32f. --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index c58a3fe7f..3490e59a5 100644 --- a/optimade.rst +++ b/optimade.rst @@ -379,10 +379,10 @@ Visiting this URL in a web browser gives a human-readable description of how to API implementations SHOULD NOT make up and use new prefixes without first getting them registered in the official list. -Database-Provider-Specific fields ---------------------------------- +Database-specific fields +~~~~~~~~~~~~~~~~~~~~~~~~ -Database-Provider-Specific fields MUST be prefixed with (one of) the prefix(es) registered by the database provider enclosed by underscores("\_", ASCII 95 dec (0x5F)). +Database-specific fields MUST be prefixed with (one of) the prefix(es) registered by the database provider enclosed by underscores("\_", ASCII 95 dec (0x5F)). **Examples**: From 9e64a3afece0d9a7c526a4882873e1edfbd694e4 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 25 Apr 2023 18:40:07 +0200 Subject: [PATCH 09/39] Revert "Add specifically that meta suffix must not be used other than for metadata fields." This reverts commit d343a86b31f494ff558851a5a8bd823977fd1b11. --- optimade.rst | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/optimade.rst b/optimade.rst index 3490e59a5..de2d04e03 100644 --- a/optimade.rst +++ b/optimade.rst @@ -379,20 +379,13 @@ Visiting this URL in a web browser gives a human-readable description of how to API implementations SHOULD NOT make up and use new prefixes without first getting them registered in the official list. -Database-specific fields -~~~~~~~~~~~~~~~~~~~~~~~~ - -Database-specific fields MUST be prefixed with (one of) the prefix(es) registered by the database provider enclosed by underscores("\_", ASCII 95 dec (0x5F)). - **Examples**: -- A database-provider-specific prefix: ``exmpl``. Used in a field name in a response: :field:`_exmpl_custom_field`. +- A database-provider-specific prefix: ``exmpl``. Used as a field name in a response: :field:`_exmpl_custom_field`. The initial underscore indicates an identifier that is under a separate namespace under the ownership of that organization. Identifiers prefixed with underscores will not be used for standardized names. -The suffix _meta is reserved for properties that contain the metadata, as described in the section `Per property metadata fields`_ and MUST not be used for fields that do not follow the guidelines described there. - URL Encoding ------------ From 6cfe92f0b52a2e094a213703eceb7e7948364723 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 25 Apr 2023 19:12:30 +0200 Subject: [PATCH 10/39] Moved mention that suffix '_meta' should not be used for regular fields to section database-provider-specific-properties. --- optimade.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/optimade.rst b/optimade.rst index 977279bfd..03041bdde 100644 --- a/optimade.rst +++ b/optimade.rst @@ -2342,6 +2342,7 @@ database-provider-specific properties Implementations are thus allowed to decide that some of these properties are part of what can be seen as the default value of :query-param:`response_fields` when that query parameter is omitted. Implementations SHOULD NOT include database-provider-specific properties when the query parameter :query-param:`response_fields` is present but does not list them. - These MUST be prefixed by a database-provider-specific prefix (see appendix `Database-Provider-Specific Namespace Prefixes`_). + - The suffix `_meta` is reserved for properties that contain metadata, as described in the section `Per property metadata fields`_ and MUST not be used for fields that do not follow the guidelines described there. - Implementations MUST add the properties to the list of :property:`properties` under the respective entry listing :endpoint:`info` endpoint (see `Entry Listing Info Endpoints`_). - **Examples**: A few examples of valid database-provided-specific property names follows: From 1414c5536e8a8f38693c991479607651e54fb8f4 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 26 Apr 2023 11:59:46 +0200 Subject: [PATCH 11/39] Add extra single quotes around the '_meta' suffix. --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 03041bdde..1f0e94294 100644 --- a/optimade.rst +++ b/optimade.rst @@ -2342,7 +2342,7 @@ database-provider-specific properties Implementations are thus allowed to decide that some of these properties are part of what can be seen as the default value of :query-param:`response_fields` when that query parameter is omitted. Implementations SHOULD NOT include database-provider-specific properties when the query parameter :query-param:`response_fields` is present but does not list them. - These MUST be prefixed by a database-provider-specific prefix (see appendix `Database-Provider-Specific Namespace Prefixes`_). - - The suffix `_meta` is reserved for properties that contain metadata, as described in the section `Per property metadata fields`_ and MUST not be used for fields that do not follow the guidelines described there. + - The suffix ``_meta`` is reserved for properties that contain metadata, as described in the section `Per property metadata fields`_ and MUST not be used for fields that do not follow the guidelines described there. - Implementations MUST add the properties to the list of :property:`properties` under the respective entry listing :endpoint:`info` endpoint (see `Entry Listing Info Endpoints`_). - **Examples**: A few examples of valid database-provided-specific property names follows: From 81f2f96b16fb0eda032102848a5d20c6f129a2f6 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 26 Apr 2023 16:54:18 +0200 Subject: [PATCH 12/39] Added example of a property definition for a metadata field. --- optimade.rst | 74 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 71 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index 1f0e94294..9c113cdab 100644 --- a/optimade.rst +++ b/optimade.rst @@ -453,8 +453,9 @@ This metadata field consists of a dictionary which MAY contain database specific Fields in this metadata MAY also have a metadata field. If an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. -The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_. -If two metadata fields have a sub field, that is present in both meta data fields, both these sub fields SHOULD have the same $id. +The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_ just as regular fields. +If a subfield is present in multiple metadata fields these subfields should have a separate entry under each of these metadata fields. +The subfields SHOULD have the same value for the $id field if the $id field is present and the subfields are otherwise identical. It SHOULD contain the field : @@ -469,13 +470,80 @@ Example of a returned metadata field: "element_ratios_meta": { "metadata_for": "element_ratios", "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], - "_exmpl_confidence_interval_meta":{ + "_exmpl_confidence_interval_meta": { "_exmpl_confidence_level": 0.95, "metadata_for": "_exmpl_confidence_interval", } //... } +Example of the property definition of a metadata field: + + .. code:: jsonc + { + "element_ratios_meta": { + "$id": "https://properties.example.com/v1.2.0/element_ratios_meta", + "title": "Metadata for the element_ratios field", + "description": "This field contains the metadata for the element_ratios field that is specific to each individual entry.", + "x-optimade-property": { + "property-format": "1.2" + }, + "x-optimade-type": "dictionary", + "x-optimade-unit": "inapplicable", + "type": ["object", "null"], + "properties" : { + "metadata_for": { + "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/metadata_for", + "description" : "Defines the property for which this metadata field contains metadata.", + "x-optimade-type": "string", + "x-optimade-unit" : "inapplicable", + "type": "string" + }, + "_exmpl_confidence_interval": { + "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_confidence_interval", + "description" : "This field consists of a list with a list that contains the lower and upper bounds of the confidence interval.", + "x-optimade-type": "list", + "x-optimade-unit" : "inapplicable", + "type": "array", + "items": { + "x-optimade-type": "list", + "description" : "The first value in this list is the lower bound for the confidence interval the second value is the upper bound.", + "x-optimade-unit" : "inapplicable", + "type": "array", + "items": { + "x-optimade-type": "float", + "x-optimade-unit" : "dimensionless", + "type": "number" + } + } + }, + "_exmpl_confidence_interval_meta": { + "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_confidence_interval_meta", + "title": "Metadata for the _exmpl_confidence_interval field", + "x-optimade-type": "dictionary", + "x-optimade-unit": "inapplicable", + "type": ["object", "null"], + "properties" : { + "metadata_for": { + "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/metadata_for", + "description" : "Defines the property for which this metadata field contains metadata.", + "x-optimade-type": "string", + "x-optimade-unit" : "inapplicable", + "type": "string" + }, + "_exmpl_confidence_level": { + "x-optimade-type": "float", + "description" : "The confidence level for this interval e.g. 0.95", + "maximum": 1.0, + "minimum": 0.0, + "x-optimade-unit" : "dimensionless", + "type": "number" + } + } + } + } + } + } Responses ========= From b69b0e8025ac3897fa3a149cb05739d88dab3014 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 1 May 2023 15:22:54 +0200 Subject: [PATCH 13/39] Update optimade.rst Co-authored-by: Rickard Armiento --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 9c113cdab..23efe3882 100644 --- a/optimade.rst +++ b/optimade.rst @@ -455,7 +455,7 @@ If an implementation supports the metadata field, it SHOULD return the metadata The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_ just as regular fields. If a subfield is present in multiple metadata fields these subfields should have a separate entry under each of these metadata fields. -The subfields SHOULD have the same value for the $id field if the $id field is present and the subfields are otherwise identical. +The subfields SHOULD have the same value for the :field:`$id` field if the :field:`$id` field is present and the subfields are otherwise identical. It SHOULD contain the field : From 478a5728fcaa425fedf6b302ddeb283b5d5584a4 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 1 May 2023 15:34:43 +0200 Subject: [PATCH 14/39] Clarifined nested meta data fields a bit. --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 9c113cdab..1893fbb9c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -450,7 +450,7 @@ The name of the metadata field consists of the name of the property for which it For example, when the field is :property:`cartesian_site_positions` the metadata field would be :field:`cartesian_site_positions_meta`. This metadata field consists of a dictionary which MAY contain database specific fields. -Fields in this metadata MAY also have a metadata field. +Fields in this metadata dictionary MAY also have their own metadata field. If an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_ just as regular fields. From 8ed79b5f8c4236fb1473e6d2ddeb5b664a1dc60c Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 1 May 2023 16:37:08 +0200 Subject: [PATCH 15/39] Placed meta_data_for field in the property definition. --- optimade.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index 1bdf741f3..8c35054f0 100644 --- a/optimade.rst +++ b/optimade.rst @@ -456,10 +456,10 @@ If an implementation supports the metadata field, it SHOULD return the metadata The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_ just as regular fields. If a subfield is present in multiple metadata fields these subfields should have a separate entry under each of these metadata fields. The subfields SHOULD have the same value for the :field:`$id` field if the :field:`$id` field is present and the subfields are otherwise identical. +In addition to the fields described in the section `property definitions`_ the property definition of a metadata field SHOULD have the key: -It SHOULD contain the field : - -- **metadata_for**: which contains a string with the name of the property for which this meta data property contains the meta data, +- **x-optimade-metadata-for**: This key contains a string with the name of the property for which this metadata property contains the metadata. + It MUST be located at the top level of the property definition. Example of a returned metadata field: @@ -468,11 +468,9 @@ Example of a returned metadata field: { "element_ratios":[0.33336, 0.22229, 0.44425], "element_ratios_meta": { - "metadata_for": "element_ratios", "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], "_exmpl_confidence_interval_meta": { "_exmpl_confidence_level": 0.95, - "metadata_for": "_exmpl_confidence_interval", } //... } @@ -484,6 +482,7 @@ Example of the property definition of a metadata field: "element_ratios_meta": { "$id": "https://properties.example.com/v1.2.0/element_ratios_meta", "title": "Metadata for the element_ratios field", + "x-optimade-metadata-for": "element_ratios", "description": "This field contains the metadata for the element_ratios field that is specific to each individual entry.", "x-optimade-property": { "property-format": "1.2" @@ -520,6 +519,7 @@ Example of the property definition of a metadata field: "_exmpl_confidence_interval_meta": { "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_confidence_interval_meta", "title": "Metadata for the _exmpl_confidence_interval field", + "x-optimade-metadata-for": "_exmpl_confidence_interval", "x-optimade-type": "dictionary", "x-optimade-unit": "inapplicable", "type": ["object", "null"], From f1e179c3a814b6f46a0d22f48de315251c18a917 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 4 May 2023 13:08:38 +0200 Subject: [PATCH 16/39] Removed property definition metadata_for field as it is now part of the property definition itself. --- optimade.rst | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/optimade.rst b/optimade.rst index 8c35054f0..c396883e2 100644 --- a/optimade.rst +++ b/optimade.rst @@ -491,13 +491,6 @@ Example of the property definition of a metadata field: "x-optimade-unit": "inapplicable", "type": ["object", "null"], "properties" : { - "metadata_for": { - "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/metadata_for", - "description" : "Defines the property for which this metadata field contains metadata.", - "x-optimade-type": "string", - "x-optimade-unit" : "inapplicable", - "type": "string" - }, "_exmpl_confidence_interval": { "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_confidence_interval", "description" : "This field consists of a list with a list that contains the lower and upper bounds of the confidence interval.", @@ -524,13 +517,6 @@ Example of the property definition of a metadata field: "x-optimade-unit": "inapplicable", "type": ["object", "null"], "properties" : { - "metadata_for": { - "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/metadata_for", - "description" : "Defines the property for which this metadata field contains metadata.", - "x-optimade-type": "string", - "x-optimade-unit" : "inapplicable", - "type": "string" - }, "_exmpl_confidence_level": { "x-optimade-type": "float", "description" : "The confidence level for this interval e.g. 0.95", From a60eb5c22f7f18832cf52dbcbd9534ac5bf7698e Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 4 May 2023 13:30:58 +0200 Subject: [PATCH 17/39] Added comma to improve readability. --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index c396883e2..5380ac253 100644 --- a/optimade.rst +++ b/optimade.rst @@ -456,7 +456,7 @@ If an implementation supports the metadata field, it SHOULD return the metadata The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_ just as regular fields. If a subfield is present in multiple metadata fields these subfields should have a separate entry under each of these metadata fields. The subfields SHOULD have the same value for the :field:`$id` field if the :field:`$id` field is present and the subfields are otherwise identical. -In addition to the fields described in the section `property definitions`_ the property definition of a metadata field SHOULD have the key: +In addition to the fields described in the section `property definitions`_, the property definition of a metadata field SHOULD have the key: - **x-optimade-metadata-for**: This key contains a string with the name of the property for which this metadata property contains the metadata. It MUST be located at the top level of the property definition. From df401729ada202d621bf3a48d4738c791e023b73 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 4 May 2023 13:30:58 +0200 Subject: [PATCH 18/39] Added comma to improve readability. --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index c396883e2..f70558f46 100644 --- a/optimade.rst +++ b/optimade.rst @@ -454,9 +454,9 @@ Fields in this metadata dictionary MAY also have their own metadata field. If an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_ just as regular fields. -If a subfield is present in multiple metadata fields these subfields should have a separate entry under each of these metadata fields. +If a subfield is present in multiple metadata fields, these subfields should have a separate entry under each of these metadata fields. The subfields SHOULD have the same value for the :field:`$id` field if the :field:`$id` field is present and the subfields are otherwise identical. -In addition to the fields described in the section `property definitions`_ the property definition of a metadata field SHOULD have the key: +In addition to the fields described in the section `property definitions`_, the property definition of a metadata field SHOULD have the key: - **x-optimade-metadata-for**: This key contains a string with the name of the property for which this metadata property contains the metadata. It MUST be located at the top level of the property definition. From 1e81988c498ada92dd0d16d3822d05c043b17c8e Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 4 May 2023 18:15:20 +0200 Subject: [PATCH 19/39] Postponed decision on whether or not nested metadata fields are allowed --- optimade.rst | 32 ++++++++++---------------------- 1 file changed, 10 insertions(+), 22 deletions(-) diff --git a/optimade.rst b/optimade.rst index f70558f46..5a7fd1f5c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -450,7 +450,6 @@ The name of the metadata field consists of the name of the property for which it For example, when the field is :property:`cartesian_site_positions` the metadata field would be :field:`cartesian_site_positions_meta`. This metadata field consists of a dictionary which MAY contain database specific fields. -Fields in this metadata dictionary MAY also have their own metadata field. If an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_ just as regular fields. @@ -469,16 +468,15 @@ Example of a returned metadata field: "element_ratios":[0.33336, 0.22229, 0.44425], "element_ratios_meta": { "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], - "_exmpl_confidence_interval_meta": { - "_exmpl_confidence_level": 0.95, - } + "_exmpl_confidence_level": 0.95, + } //... } Example of the property definition of a metadata field: .. code:: jsonc - { + { "element_ratios_meta": { "$id": "https://properties.example.com/v1.2.0/element_ratios_meta", "title": "Metadata for the element_ratios field", @@ -509,23 +507,13 @@ Example of the property definition of a metadata field: } } }, - "_exmpl_confidence_interval_meta": { - "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_confidence_interval_meta", - "title": "Metadata for the _exmpl_confidence_interval field", - "x-optimade-metadata-for": "_exmpl_confidence_interval", - "x-optimade-type": "dictionary", - "x-optimade-unit": "inapplicable", - "type": ["object", "null"], - "properties" : { - "_exmpl_confidence_level": { - "x-optimade-type": "float", - "description" : "The confidence level for this interval e.g. 0.95", - "maximum": 1.0, - "minimum": 0.0, - "x-optimade-unit" : "dimensionless", - "type": "number" - } - } + "_exmpl_confidence_level": { + "x-optimade-type": "float", + "description" : "The confidence level for this interval e.g. 0.95", + "maximum": 1.0, + "minimum": 0.0, + "x-optimade-unit" : "dimensionless", + "type": "number" } } } From 6f56a17f01bd9d2ab2249d454c613989a4c2ba5e Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 1 Jun 2023 13:26:06 +0200 Subject: [PATCH 20/39] Moved the metadata fields to the JSON API per entry meta field as suggested by Rickard in https://github.com/JPBergsma/OPTIMADE/pull/2/files. Co-authored-by: Rickard Armiento --- optimade.rst | 92 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 63 insertions(+), 29 deletions(-) diff --git a/optimade.rst b/optimade.rst index 5a7fd1f5c..90ad58097 100644 --- a/optimade.rst +++ b/optimade.rst @@ -442,42 +442,64 @@ For example, the following query can be sent to API implementations `exmpl1` and :filter:`filter=_exmpl1_band_gap<2.0 OR _exmpl2_band_gap<2.5` -Per property metadata fields ----------------------------- - -Implementations are allowed to specify an OPTIONAL field containing per entry metadata for a property. -The name of the metadata field consists of the name of the property for which it contains the metadata, suffixed with "_meta". -For example, when the field is :property:`cartesian_site_positions` the metadata field would be :field:`cartesian_site_positions_meta`. +Metadata properties +------------------- -This metadata field consists of a dictionary which MAY contain database specific fields. -If an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. +A metadata property represents entry and property-specific metadata for a given entry. +How these are communicated in the response depends on the response format. +For the JSON response format, the resource object metadata field, :field:`meta`, is used, see `JSON Response Schema: Common Fields`_. -The metadata fields and their subfields should be described in the property definitions as described in the section `property definitions`_ just as regular fields. -If a subfield is present in multiple metadata fields, these subfields should have a separate entry under each of these metadata fields. -The subfields SHOULD have the same value for the :field:`$id` field if the :field:`$id` field is present and the subfields are otherwise identical. -In addition to the fields described in the section `property definitions`_, the property definition of a metadata field SHOULD have the key: +The metadata property is a dictionary in the format specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property Definitions`_. +Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-specific prefix even for metadata for database-specific fields. +For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. +The reason for this limitation is to avoid name collisions with standard metadata fields defined by the OPTIMADE standard that apply also to database-specific data fields. -- **x-optimade-metadata-for**: This key contains a string with the name of the property for which this metadata property contains the metadata. - It MUST be located at the top level of the property definition. +Implementation of the :field:`meta` field is OPTIONAL. +However, when an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. -Example of a returned metadata field: +Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database specific per entry metadata field :field:_exmpl_uploader` : - .. code:: jsonc - - { - "element_ratios":[0.33336, 0.22229, 0.44425], - "element_ratios_meta": { - "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], - "_exmpl_confidence_level": 0.95, +.. code:: jsonc + { + "data": [ + { + "type": "structures", + "id": "example.db:structs:0001", + "attributes": { + "element_ratios":[0.33336, 0.22229, 0.44425], + }, + "meta": { + "element_ratios": { + "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], + "_exmpl_confidence_level": 0.95 + }, + "_exmpl_uploader" : "John Doe" + } + }, + { + "type": "structures", + "id": "example.db:structs:1234", + "attributes": { + "element_ratios":[0.5, 0.5], + }, + "meta": { + "element_ratios": { + "_exmpl_confidence_interval": [[0.6,0.4],[0.7,0.4]], + "_exmpl_confidence_level": 0.90 + }, + "_exmpl_uploader" : "John Doe" + } } //... - } + ] + // ... + } -Example of the property definition of a metadata field: +Example of the corresponding metadata property definition contained in the field :field:`x-optimade-metadata-definition` which is placed in the property definition of :field:`element_ratios`: .. code:: jsonc - { - "element_ratios_meta": { + // ... + "x-optimade-metadata-definition": { "$id": "https://properties.example.com/v1.2.0/element_ratios_meta", "title": "Metadata for the element_ratios field", "x-optimade-metadata-for": "element_ratios", @@ -517,7 +539,7 @@ Example of the property definition of a metadata field: } } } - } + // ... Responses ========= @@ -669,6 +691,10 @@ Every response SHOULD contain the following fields, and MUST contain at least :f - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON API resource object `__ or a *list* of JSON API resource objects. Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. + Each resource object MAY contain a :field:`meta` field used to communicate `Metadata properties`_. + The keys of the :field:`meta` field are the names of the fields in :field:`attributes` for which metadata is available. + The values belonging to these keys are dictionaries containing the relevant metadata fields.. + In addition the :field:`meta` can also hold metadata fields that apply to the entry as a whole. The response MAY also return resources related to the primary data in the field: @@ -992,7 +1018,10 @@ OPTIONALLY it can also contain the following fields: - **self**: the entry's URL -- **meta**: a `JSON API meta object `__ that contains non-standard meta-information about the object. +- **meta**: a `JSON API meta object `__ that is used to communicate `Metadata properties`_. + The keys of the :field:`meta` field are the names of the fields in :field:`attributes` for which metadata is available. + The values belonging to these keys are dictionaries containing the relevant metadata fields.. + In addition the :field:`meta` can also hold metadata fields that apply to the entry as a whole. - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON API Relationships `__. The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON API resource identifier object. @@ -1946,6 +1975,12 @@ A Property Definition MUST be composed according to the combination of the requi - :field:`uri`: String. A URI of the external resource (which MAY be a resolvable URL). +**OPTIONAL keys for the outermost level of the Property Definition:** + + - :field:`x-optimade-metadata-definition`: Dictionary. + A dictionary that is itself a property definition, which defines the format of the metadata dictionary for this field. + The :field:`x-optimade-metadata-definition` field SHOULD not include another :field:`x-optimade-metadata-definition` field. + **REQUIRED keys for all levels of the Property Definition:** - :field:`x-optimade-type`: String @@ -2384,7 +2419,6 @@ database-provider-specific properties Implementations are thus allowed to decide that some of these properties are part of what can be seen as the default value of :query-param:`response_fields` when that query parameter is omitted. Implementations SHOULD NOT include database-provider-specific properties when the query parameter :query-param:`response_fields` is present but does not list them. - These MUST be prefixed by a database-provider-specific prefix (see appendix `Database-Provider-Specific Namespace Prefixes`_). - - The suffix ``_meta`` is reserved for properties that contain metadata, as described in the section `Per property metadata fields`_ and MUST not be used for fields that do not follow the guidelines described there. - Implementations MUST add the properties to the list of :property:`properties` under the respective entry listing :endpoint:`info` endpoint (see `Entry Listing Info Endpoints`_). - **Examples**: A few examples of valid database-provided-specific property names follows: From 97a04359f9d4f51f31e43299e749cc39514d7227 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 1 Jun 2023 15:22:53 +0200 Subject: [PATCH 21/39] Removed x-optimade-metadata-for from example property declaration as the explanation for this field had already been removed by rartino. --- optimade.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 90ad58097..6d76672f8 100644 --- a/optimade.rst +++ b/optimade.rst @@ -502,7 +502,6 @@ Example of the corresponding metadata property definition contained in the field "x-optimade-metadata-definition": { "$id": "https://properties.example.com/v1.2.0/element_ratios_meta", "title": "Metadata for the element_ratios field", - "x-optimade-metadata-for": "element_ratios", "description": "This field contains the metadata for the element_ratios field that is specific to each individual entry.", "x-optimade-property": { "property-format": "1.2" From 00a7a08d137d7087a0b86934301a89237c448146 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 1 Jun 2023 16:04:20 +0200 Subject: [PATCH 22/39] Tried to make it clearer when a server SHOULD return the meta data. --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 6d76672f8..b76a95bfa 100644 --- a/optimade.rst +++ b/optimade.rst @@ -455,7 +455,7 @@ For example, the metadata property definition of the field :field:`_exmpl_exampl The reason for this limitation is to avoid name collisions with standard metadata fields defined by the OPTIMADE standard that apply also to database-specific data fields. Implementation of the :field:`meta` field is OPTIONAL. -However, when an implementation supports the metadata field, it SHOULD return the metadata field whenever the property to which the metadata field belongs is returned. +However, when an implementation supports the :field:`meta` field, it SHOULD return the metadata for any of the fields that are in the returned entries as well as metadata that applies to the entries as a whole. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database specific per entry metadata field :field:_exmpl_uploader` : From 7099a69086e717ef82793fd4c2578d796fcef830 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 2 Jun 2023 19:03:24 +0200 Subject: [PATCH 23/39] removed extra '.' --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index b76a95bfa..ce5ba8476 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1019,7 +1019,7 @@ OPTIONALLY it can also contain the following fields: - **meta**: a `JSON API meta object `__ that is used to communicate `Metadata properties`_. The keys of the :field:`meta` field are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields.. + The values belonging to these keys are dictionaries containing the relevant metadata fields. In addition the :field:`meta` can also hold metadata fields that apply to the entry as a whole. - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON API Relationships `__. From 09fffea40e40bd8090b72109c4669aac3b8e57e0 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 8 Jun 2023 23:44:06 +0200 Subject: [PATCH 24/39] Apply suggestions from review --- optimade.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index ce5ba8476..7fa7719dd 100644 --- a/optimade.rst +++ b/optimade.rst @@ -447,7 +447,7 @@ Metadata properties A metadata property represents entry and property-specific metadata for a given entry. How these are communicated in the response depends on the response format. -For the JSON response format, the resource object metadata field, :field:`meta`, is used, see `JSON Response Schema: Common Fields`_. +For the JSON response format, a subfield :field:`property_metadata` of the resource object metadata field, :field:`meta`, is used, see `JSON Response Schema: Common Fields`_. The metadata property is a dictionary in the format specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property Definitions`_. Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-specific prefix even for metadata for database-specific fields. @@ -469,11 +469,13 @@ Example of a response in the JSON response format with two structure entries tha "element_ratios":[0.33336, 0.22229, 0.44425], }, "meta": { - "element_ratios": { - "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], - "_exmpl_confidence_level": 0.95 - }, - "_exmpl_uploader" : "John Doe" + "property_metadata": { + "element_ratios": { + "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], + "_exmpl_confidence_level": 0.95 + }, + "_exmpl_uploader" : "John Doe" + } } }, { From cceabb51bd7318f2044df375122eebf2c04a7066 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 9 Jun 2023 00:51:35 +0200 Subject: [PATCH 25/39] Fix where metadata properties appear based on discussion --- optimade.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/optimade.rst b/optimade.rst index 7fa7719dd..33a6a0a07 100644 --- a/optimade.rst +++ b/optimade.rst @@ -692,10 +692,10 @@ Every response SHOULD contain the following fields, and MUST contain at least :f - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON API resource object `__ or a *list* of JSON API resource objects. Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. - Each resource object MAY contain a :field:`meta` field used to communicate `Metadata properties`_. - The keys of the :field:`meta` field are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields.. - In addition the :field:`meta` can also hold metadata fields that apply to the entry as a whole. + Each resource object MAY contain a :field:`meta` field used to communicate metadata. + The subkey :field:`property_metadata` is used to communicate `Metadata properties`_. + The keys of the :field:`property_metadata` field are the names of the fields in :field:`attributes` for which metadata is available. + The values belonging to these keys are dictionaries containing the relevant metadata fields. The response MAY also return resources related to the primary data in the field: @@ -1019,10 +1019,12 @@ OPTIONALLY it can also contain the following fields: - **self**: the entry's URL -- **meta**: a `JSON API meta object `__ that is used to communicate `Metadata properties`_. - The keys of the :field:`meta` field are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields. - In addition the :field:`meta` can also hold metadata fields that apply to the entry as a whole. +- **meta**: a `JSON API meta object `__ that is used to communicate per-entry metadata. + It MAY contain the following keys: + + - **property_metadata**: an object used to communicate `Metadata properties`_. + The keys are the names of the fields in :field:`attributes` for which metadata is available. + The values belonging to these keys are dictionaries containing the relevant metadata fields. - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON API Relationships `__. The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON API resource identifier object. From 45b98b88b7b25f84f268603130525aa7144381e8 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 9 Jun 2023 01:02:14 +0200 Subject: [PATCH 26/39] Remove the repetition of info about the data->meta field --- optimade.rst | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/optimade.rst b/optimade.rst index 33a6a0a07..10bb71055 100644 --- a/optimade.rst +++ b/optimade.rst @@ -692,10 +692,11 @@ Every response SHOULD contain the following fields, and MUST contain at least :f - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON API resource object `__ or a *list* of JSON API resource objects. Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. - Each resource object MAY contain a :field:`meta` field used to communicate metadata. - The subkey :field:`property_metadata` is used to communicate `Metadata properties`_. - The keys of the :field:`property_metadata` field are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields. + The :field:`data` field MAY also contain a :field:`meta` field with the following key: + + - **property_metadata**: an object containing per-entry and per-property metadata. + The keys are the names of the fields in :field:`attributes` for which metadata is available. + The values belonging to these keys are dictionaries containing the relevant metadata fields. The response MAY also return resources related to the primary data in the field: @@ -1019,12 +1020,8 @@ OPTIONALLY it can also contain the following fields: - **self**: the entry's URL -- **meta**: a `JSON API meta object `__ that is used to communicate per-entry metadata. - It MAY contain the following keys: - - - **property_metadata**: an object used to communicate `Metadata properties`_. - The keys are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields. +- **meta**: a `JSON API meta object `__ that is used to communicate metadata. + See `JSON Response Schema: Common Fields`_ for more information about this field. - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON API Relationships `__. The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON API resource identifier object. From b95c4654b7347add5e6b903da926a97536386447 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sun, 11 Jun 2023 14:27:13 +0200 Subject: [PATCH 27/39] Provide an example that is less objectable for metadata --- optimade.rst | 34 +++++----------------------------- 1 file changed, 5 insertions(+), 29 deletions(-) diff --git a/optimade.rst b/optimade.rst index 10bb71055..b34a45617 100644 --- a/optimade.rst +++ b/optimade.rst @@ -502,41 +502,17 @@ Example of the corresponding metadata property definition contained in the field .. code:: jsonc // ... "x-optimade-metadata-definition": { - "$id": "https://properties.example.com/v1.2.0/element_ratios_meta", "title": "Metadata for the element_ratios field", - "description": "This field contains the metadata for the element_ratios field that is specific to each individual entry.", - "x-optimade-property": { - "property-format": "1.2" - }, + "description": "This field contains the per-entry metadata for the element_ratios field.", "x-optimade-type": "dictionary", "x-optimade-unit": "inapplicable", "type": ["object", "null"], "properties" : { - "_exmpl_confidence_interval": { - "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_confidence_interval", - "description" : "This field consists of a list with a list that contains the lower and upper bounds of the confidence interval.", - "x-optimade-type": "list", + "_exmpl_originates_from_project": { + "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_originates_from_project", + "description" : "A string naming the internal example.com project id where this property was added to the database.", + "x-optimade-type": "string", "x-optimade-unit" : "inapplicable", - "type": "array", - "items": { - "x-optimade-type": "list", - "description" : "The first value in this list is the lower bound for the confidence interval the second value is the upper bound.", - "x-optimade-unit" : "inapplicable", - "type": "array", - "items": { - "x-optimade-type": "float", - "x-optimade-unit" : "dimensionless", - "type": "number" - } - } - }, - "_exmpl_confidence_level": { - "x-optimade-type": "float", - "description" : "The confidence level for this interval e.g. 0.95", - "maximum": 1.0, - "minimum": 0.0, - "x-optimade-unit" : "dimensionless", - "type": "number" } } } From 134a9065fd7cd433249b461496787a7f827069cf Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sun, 11 Jun 2023 14:30:41 +0200 Subject: [PATCH 28/39] Fix examples as requested by workshop --- optimade.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index b34a45617..7510805ca 100644 --- a/optimade.rst +++ b/optimade.rst @@ -471,10 +471,8 @@ Example of a response in the JSON response format with two structure entries tha "meta": { "property_metadata": { "element_ratios": { - "_exmpl_confidence_interval": [[0.33325,0.33347],[0.22190,0.22268],[0.44392,0.44458]], - "_exmpl_confidence_level": 0.95 + "_exmpl_originates_from_project": "piezoelectic_perovskites" }, - "_exmpl_uploader" : "John Doe" } } }, @@ -486,10 +484,8 @@ Example of a response in the JSON response format with two structure entries tha }, "meta": { "element_ratios": { - "_exmpl_confidence_interval": [[0.6,0.4],[0.7,0.4]], - "_exmpl_confidence_level": 0.90 + "_exmpl_originates_from_project": "ferroelectric_binaries" }, - "_exmpl_uploader" : "John Doe" } } //... From 078f1743c3009ca073f6f10109faa29feaab4277 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 12 Jun 2023 16:04:49 +0200 Subject: [PATCH 29/39] Apply suggestions from code review Co-authored-by: Andrius Merkys --- optimade.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index 7510805ca..f354ef21e 100644 --- a/optimade.rst +++ b/optimade.rst @@ -483,9 +483,11 @@ Example of a response in the JSON response format with two structure entries tha "element_ratios":[0.5, 0.5], }, "meta": { - "element_ratios": { - "_exmpl_originates_from_project": "ferroelectric_binaries" - }, + "propertiy_metadata":{ + "element_ratios": { + "_exmpl_originates_from_project": "ferroelectric_binaries" + }, + } } } //... @@ -664,7 +666,7 @@ Every response SHOULD contain the following fields, and MUST contain at least :f - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON API resource object `__ or a *list* of JSON API resource objects. Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. - The :field:`data` field MAY also contain a :field:`meta` field with the following key: + Every resource object MAY also contain a :field:`meta` field with the following key: - **property_metadata**: an object containing per-entry and per-property metadata. The keys are the names of the fields in :field:`attributes` for which metadata is available. From e15aaff896a905d728c2ce69d314898df7290263 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 12 Jun 2023 17:20:09 +0200 Subject: [PATCH 30/39] Apply suggestions from review Co-authored-by: Antanas Vaitkus --- optimade.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index f354ef21e..2c612ba38 100644 --- a/optimade.rst +++ b/optimade.rst @@ -457,7 +457,7 @@ The reason for this limitation is to avoid name collisions with standard metadat Implementation of the :field:`meta` field is OPTIONAL. However, when an implementation supports the :field:`meta` field, it SHOULD return the metadata for any of the fields that are in the returned entries as well as metadata that applies to the entries as a whole. -Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database specific per entry metadata field :field:_exmpl_uploader` : +Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database specific per entry metadata field :field:`_exmpl_uploader` : .. code:: jsonc { @@ -466,13 +466,13 @@ Example of a response in the JSON response format with two structure entries tha "type": "structures", "id": "example.db:structs:0001", "attributes": { - "element_ratios":[0.33336, 0.22229, 0.44425], + "element_ratios":[0.33336, 0.22229, 0.44425] }, "meta": { "property_metadata": { "element_ratios": { "_exmpl_originates_from_project": "piezoelectic_perovskites" - }, + } } } }, @@ -480,13 +480,13 @@ Example of a response in the JSON response format with two structure entries tha "type": "structures", "id": "example.db:structs:1234", "attributes": { - "element_ratios":[0.5, 0.5], + "element_ratios":[0.5, 0.5] }, "meta": { "propertiy_metadata":{ "element_ratios": { "_exmpl_originates_from_project": "ferroelectric_binaries" - }, + } } } } @@ -510,7 +510,7 @@ Example of the corresponding metadata property definition contained in the field "$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_originates_from_project", "description" : "A string naming the internal example.com project id where this property was added to the database.", "x-optimade-type": "string", - "x-optimade-unit" : "inapplicable", + "x-optimade-unit" : "inapplicable" } } } From 9c4417f6438562b560111c6c4deb532f99b34ee4 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 12 Jun 2023 17:21:52 +0200 Subject: [PATCH 31/39] Fix JSON example spelling error --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 2c612ba38..5746f7f02 100644 --- a/optimade.rst +++ b/optimade.rst @@ -483,7 +483,7 @@ Example of a response in the JSON response format with two structure entries tha "element_ratios":[0.5, 0.5] }, "meta": { - "propertiy_metadata":{ + "property_metadata":{ "element_ratios": { "_exmpl_originates_from_project": "ferroelectric_binaries" } From b36c372ee4aa1a838fb5d77cd586513c93d71454 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 14 Jun 2023 15:54:44 +0200 Subject: [PATCH 32/39] Update optimade.rst Add link back to the Metadata_properties section. --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 5746f7f02..0369082fd 100644 --- a/optimade.rst +++ b/optimade.rst @@ -670,7 +670,7 @@ Every response SHOULD contain the following fields, and MUST contain at least :f - **property_metadata**: an object containing per-entry and per-property metadata. The keys are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields. + The values belonging to these keys are dictionaries containing the relevant metadata fields. See also `Metadata properties`_ The response MAY also return resources related to the primary data in the field: From cc42d8f62c747391d05a7285bc460ee3b9e7f615 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Wed, 14 Jun 2023 17:24:40 +0200 Subject: [PATCH 33/39] Apply suggestions from review Co-authored-by: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> --- optimade.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index 0369082fd..65b7e108a 100644 --- a/optimade.rst +++ b/optimade.rst @@ -447,15 +447,16 @@ Metadata properties A metadata property represents entry and property-specific metadata for a given entry. How these are communicated in the response depends on the response format. -For the JSON response format, a subfield :field:`property_metadata` of the resource object metadata field, :field:`meta`, is used, see `JSON Response Schema: Common Fields`_. +For the JSON response format, the metadata properties are stored in the resource object metadata field, :field:`meta`, see `JSON Response Schema: Common Fields`_. +Metadata, that does not apply to the entry as a whole but only to the individual properties under attributes, is stored under the subfield :field:`property_metadata` in dictionaries, here called metadata properties, with the name of the property as the key . -The metadata property is a dictionary in the format specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property Definitions`_. +The format of the metadata property is specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property #Definitions`_. Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-specific prefix even for metadata for database-specific fields. For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. -The reason for this limitation is to avoid name collisions with standard metadata fields defined by the OPTIMADE standard that apply also to database-specific data fields. +The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. Implementation of the :field:`meta` field is OPTIONAL. -However, when an implementation supports the :field:`meta` field, it SHOULD return the metadata for any of the fields that are in the returned entries as well as metadata that applies to the entries as a whole. +However, when an implementation supports the :field:`meta` field, it SHOULD return the metadata for any of the fields that are in the response as well as metadata that applies to the entries as a whole. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database specific per entry metadata field :field:`_exmpl_uploader` : From 52ad0c0c2a0adf4d00e679d2ed4f5a146b7035cc Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Wed, 14 Jun 2023 17:39:50 +0200 Subject: [PATCH 34/39] Remove global per-entry metadata properties --- optimade.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 65b7e108a..a7f06c717 100644 --- a/optimade.rst +++ b/optimade.rst @@ -447,8 +447,7 @@ Metadata properties A metadata property represents entry and property-specific metadata for a given entry. How these are communicated in the response depends on the response format. -For the JSON response format, the metadata properties are stored in the resource object metadata field, :field:`meta`, see `JSON Response Schema: Common Fields`_. -Metadata, that does not apply to the entry as a whole but only to the individual properties under attributes, is stored under the subfield :field:`property_metadata` in dictionaries, here called metadata properties, with the name of the property as the key . +For the JSON response format, the metadata properties are stored in the resource object metadata field, :field:`meta` in a dictionary field :field:`property_metadata` with the keys equal to the names of the respective properties for which metadata is available; see `JSON Response Schema: Common Fields`_. The format of the metadata property is specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property #Definitions`_. Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-specific prefix even for metadata for database-specific fields. From 62f987d160fb5d6fe56833ac60de207b8c34e21f Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Wed, 14 Jun 2023 17:42:06 +0200 Subject: [PATCH 35/39] Remove ability for per-entry global metadata --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index a7f06c717..8f8037dbf 100644 --- a/optimade.rst +++ b/optimade.rst @@ -455,7 +455,7 @@ For example, the metadata property definition of the field :field:`_exmpl_exampl The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. Implementation of the :field:`meta` field is OPTIONAL. -However, when an implementation supports the :field:`meta` field, it SHOULD return the metadata for any of the fields that are in the response as well as metadata that applies to the entries as a whole. +However, when an implementation supports the :field:`meta` field, it SHOULD return the metadata for any of the fields that are in the response. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database specific per entry metadata field :field:`_exmpl_uploader` : From aaddc82d5994bcd16b7ae82cc12153eced8ca54c Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 15 Jun 2023 13:14:46 +0200 Subject: [PATCH 36/39] Apply suggestions from review Co-authored-by: Antanas Vaitkus --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index 8f8037dbf..870c4071e 100644 --- a/optimade.rst +++ b/optimade.rst @@ -450,14 +450,14 @@ How these are communicated in the response depends on the response format. For the JSON response format, the metadata properties are stored in the resource object metadata field, :field:`meta` in a dictionary field :field:`property_metadata` with the keys equal to the names of the respective properties for which metadata is available; see `JSON Response Schema: Common Fields`_. The format of the metadata property is specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property #Definitions`_. -Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-specific prefix even for metadata for database-specific fields. +Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-provider-specific prefix even for metadata for database-specific fields. For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. Implementation of the :field:`meta` field is OPTIONAL. -However, when an implementation supports the :field:`meta` field, it SHOULD return the metadata for any of the fields that are in the response. +However, when an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all fields present in the data part of the response which has metadata. -Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database specific per entry metadata field :field:`_exmpl_uploader` : +Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database-specific per entry metadata field :field:`_exmpl_uploader` : .. code:: jsonc { From 4feb8e1bbd84b5b92cef3123bd7e0e944238001f Mon Sep 17 00:00:00 2001 From: Andrius Merkys Date: Fri, 16 Jun 2023 10:17:24 +0300 Subject: [PATCH 37/39] Update optimade.rst --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index b578c7221..018b35069 100644 --- a/optimade.rst +++ b/optimade.rst @@ -499,7 +499,7 @@ A metadata property represents entry and property-specific metadata for a given How these are communicated in the response depends on the response format. For the JSON response format, the metadata properties are stored in the resource object metadata field, :field:`meta` in a dictionary field :field:`property_metadata` with the keys equal to the names of the respective properties for which metadata is available; see `JSON Response Schema: Common Fields`_. -The format of the metadata property is specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property #Definitions`_. +The format of the metadata property is specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property Definitions`_. Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-provider-specific prefix even for metadata for database-specific fields. For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. From 699203878e83146f35e05c1279350f99d874fa50 Mon Sep 17 00:00:00 2001 From: Andrius Merkys Date: Fri, 16 Jun 2023 10:22:20 +0300 Subject: [PATCH 38/39] Cleanup trailing whitespace. --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 018b35069..caf18d1e1 100644 --- a/optimade.rst +++ b/optimade.rst @@ -721,7 +721,7 @@ Every response SHOULD contain the following fields, and MUST contain at least :f - **property_metadata**: an object containing per-entry and per-property metadata. The keys are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields. + The values belonging to these keys are dictionaries containing the relevant metadata fields. See also `Metadata properties`_ - **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. From 6ffe55030e9a71572ffca2b0604592350a0413dc Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 16 Jun 2023 12:00:00 +0200 Subject: [PATCH 39/39] Apply suggestions from review Co-authored-by: Antanas Vaitkus --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index caf18d1e1..ccfded5de 100644 --- a/optimade.rst +++ b/optimade.rst @@ -497,17 +497,17 @@ Metadata properties A metadata property represents entry and property-specific metadata for a given entry. How these are communicated in the response depends on the response format. -For the JSON response format, the metadata properties are stored in the resource object metadata field, :field:`meta` in a dictionary field :field:`property_metadata` with the keys equal to the names of the respective properties for which metadata is available; see `JSON Response Schema: Common Fields`_. +For the JSON response format, the metadata properties are stored in the resource object metadata field, :field:`meta` in a dictionary field :field:`property_metadata` with the keys equal to the names of the respective properties for which metadata is available, see `JSON Response Schema: Common Fields`_. The format of the metadata property is specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property Definitions`_. -Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-provider-specific prefix even for metadata for database-specific fields. +Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-provider-specific prefix even for metadata of database-specific fields. For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. Implementation of the :field:`meta` field is OPTIONAL. However, when an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all fields present in the data part of the response which has metadata. -Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database-specific per entry metadata field :field:`_exmpl_uploader` : +Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database-specific per entry metadata field :field:`_exmpl_originates_from_project` : .. code:: jsonc {