added unit test for "if-equal" header: "skip-minimizing-merge"

* also added documentation
* enhanced release notes + blogpost about release with last-minute addition

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

base/model/src/main/java/org/eclipse/ditto/base/model/headers/IfEqual.java

@@ -37,9 +37,10 @@ public enum IfEqual {
     /**
      * Option which will skip the update of a twin if the new value is the same (via {@code equal()}) than the value
      * before.
-     * And additionally minimized a "Merge" command to only the actually changed fields compared to the current state
+     * And additionally minimizes a "Merge" command to only the actually changed fields compared to the current state
      * of the entity. This can be beneficial to reduce (persisted and emitted) events to the minimum of what actually
      * did change.
+     *
      * @since 3.4.0
      */
     SKIP_MINIMIZING_MERGE("skip-minimizing-merge");

documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md → documentation/src/main/resources/_posts/2023-10-16-release-announcement-340.md

@@ -1,7 +1,7 @@
 ---
 title: "Announcing Eclipse Ditto Release 3.4.0"
 published: true
-permalink: 2023-10-12-release-announcement-340.html
+permalink: 2023-10-16-release-announcement-340.html
 layout: post
 author: thomas_jaeckle
 tags: [blog]
@@ -36,6 +36,7 @@ Eclipse Ditto 3.4.0 focuses on the following areas:
 * Addition of a **new placeholder** to use **in connections** to use **payload of the thing JSON** e.g. in headers or addresses
 * New **placeholder functions** for **joining** multiple elements into a single string and doing **URL-encoding and -decoding**
 * Configure **MQTT message expiry interval for published messages** via a header
+* **Reduce patch/merge thing commands** to **modify** only the **actually changed values** with a new option
 
 The following non-functional work is also included:
 

documentation/src/main/resources/jsonschema/protocol-envelope.json

@@ -34,7 +34,7 @@
         },
         "if-equal": {
           "type": "string",
-          "description": "The `if-equal` header can take the values 'update' (which is the default if omitted) or 'skip'.\nIf 'update' is defined, the entity will always be updated, even if it is equal before the update.\nIf 'skip' is defined, the entity not be updated if it is equal before the update. In this case a 'Not Modified' 304 status is returned."
+          "description": "The `if-equal` header can take the values 'update' (which is the default if omitted) or 'skip'.\nIf 'update' is defined, the entity will always be updated, even if it is equal before the update.\nIf 'skip' is defined, the entity not be updated if it is equal before the update. In this case a 'Precondition Failed' 412 status is returned.\nIf 'skip-minimizing-merge' is defined, the entity will not be updated if it is equal before the update. In this case a 'Precondition Failed' 412 status is returned. Additionally, merge/patch commands will be minimized to only the fields which actually changed, compared to the current state of the entity."
         },
         "response-required": {
           "type": "boolean",

documentation/src/main/resources/openapi/ditto-api-2.yml

@@ -8595,13 +8595,14 @@ components:
     IfEqualHeaderParam:
       name: if-equal
       in: header
-      description: 'The `if-equal` header can take the values ''update'' (which is the default if omitted) or ''skip''. If ''update'' is defined, the entity will always be updated, even if it is equal before the update. If ''skip'' is defined, the entity not be updated if it is equal before the update.  In this case a ''Not Modified'' 304 status is returned.'
+      description: 'The `if-equal` header can take the values ''update'' (which is the default if omitted), ''skip'' or ''skip-minimizing-merge''. If ''update'' is defined, the entity will always be updated, even if it is equal before the update. If ''skip'' is defined, the entity not be updated if it is equal before the update.  In this case a ''Precondition Failed'' 412 status is returned. If ''skip-minimizing-merge'' is defined, the entity will not be updated if it is equal before the update.   In this case a ''Precondition Failed'' 412 status is returned. Additionally, merge/patch commands will be minimized to only the fields which actually changed, compared to the current state of the entity.'
       required: false
       schema:
         type: string
         enum:
           - update
           - skip
+          - skip-minimizing-merge
     ImportedPolicyIdPathParam:
       name: importedPolicyId
       in: path

documentation/src/main/resources/openapi/sources/parameters/ifEqualHeaderParam.yml

@@ -11,13 +11,17 @@
 name: if-equal
 in: header
 description: >-
-  The `if-equal` header can take the values 'update' (which is the default if omitted) or 'skip'.
+  The `if-equal` header can take the values 'update' (which is the default if omitted), 'skip' or 'skip-minimizing-merge'.
   If 'update' is defined, the entity will always be updated, even if it is equal before the update.
   If 'skip' is defined, the entity not be updated if it is equal before the update. 
-  In this case a 'Not Modified' 304 status is returned.
+  In this case a 'Precondition Failed' 412 status is returned.
+  If 'skip-minimizing-merge' is defined, the entity will not be updated if it is equal before the update.  
+  In this case a 'Precondition Failed' 412 status is returned.
+  Additionally, merge/patch commands will be minimized to only the fields which actually changed, compared to the current state of the entity.
 required: false
 schema:
   type: string
   enum:
     - update
-    - skip
+    - skip
+    - skip-minimizing-merge

documentation/src/main/resources/pages/ditto/httpapi-concepts.md

@@ -622,11 +622,18 @@ The following request headers can be used to issue a conditional request:
             * for read requests, status `304 (Not Modified)` without response body, with the current entity-tag of the
               resource as `ETag` header
 * `if-equal`:
-   * The `if-equal` header can take the values `'update'` (which is the default if omitted) or `'skip'`
-   * Write the resource only
-      * in case `if-equal: 'update'` is defined - even if the entity is equal before the update.
+   * The `if-equal` header can take the values `'update'` (which is the default if omitted), `'skip'` or `'skip-minimizing-merge'`
+   * Modify/Update the resource only
+      * in case `if-equal: 'update'` is defined always updates - even if the entity is equal before the update.
       * in case `if-equal: 'skip'` is defined, the entity will not be updated if it is equal before the update.  
-        In this case a 'Not Modified' 304 status is returned.
+        In this case a 'Precondition Failed' 412 status is returned.
+      * in case `if-equal: 'skip-minimizing-merge'` is defined, the entity will not be updated if it is equal before the update.  
+        In this case a 'Precondition Failed' 412 status is returned.
+        Additionally, [Merge commands](protocol-specification-things-merge.html) will be minimized to only the fields 
+        which actually changed, compared to the current state of the entity.
+         * this reduces the part of a merge/patch command to the actually changed elements, removing non-changed elements
+         * this reduces e.g. required storage in the MongoDB by a lot, if redundant data is sent often
+         * this also reduces the event payload (e.g. emitted to subscribers) to only the actually changed parts of the thing
 
 Note that the Ditto HTTP API always provides a `strong` entity-tag in the `ETag` header, thus you will never receive a
 `weak` entity-tag (see [RFC-7232 Section 2.1](https://tools.ietf.org/html/rfc7232#section-2.1)). If you convert this

documentation/src/main/resources/pages/ditto/protocol-specification-things-merge.md

@@ -20,10 +20,10 @@ If the Thing did not yet exist, it is created.
 
 ### Command
 
-| Field     | Value                   |
-|-----------|-------------------------|
-| **topic** | `<namespace>/<thingName>/things/<channel>/commands/merge`     |
-| **path**  | `/`     |
+| Field     | Value                                                                                                                                                              |
+|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **topic** | `<namespace>/<thingName>/things/<channel>/commands/merge`                                                                                                          |
+| **path**  | `/`                                                                                                                                                                |
 | **value** | The JSON value in [JSON merge patch](https://tools.ietf.org/html/rfc7396) format that is applied to the [Thing](basic-thing.html#thing) referenced in the `topic`. |
 
 ### Response

documentation/src/main/resources/pages/ditto/protocol-specification.md

@@ -93,7 +93,7 @@ There are some pre-defined headers, which have a special meaning for Ditto:
 | `ditto-originator`              | Contains the first authorization subject of the command that caused the sending of this message. Set by Ditto.                               | `String`                                                                                                                  |
 | `if-match`                      | Has the same semantics as defined for the [HTTP API](httpapi-concepts.html#conditional-requests).                                            | `String`                                                                                                                  |
 | `if-none-match`                 | Has the same semantics as defined for the [HTTP API](httpapi-concepts.html#conditional-requests).                                            | `String`                                                                                                                  |
-| `if-equal`                      | Has the same semantics as defined for the [HTTP API](httpapi-concepts.html#conditional-requests).                                            | `String` - currently possible: \['update','skip'\] - default: `'update'`                                                  |
+| `if-equal`                      | Has the same semantics as defined for the [HTTP API](httpapi-concepts.html#conditional-requests).                                            | `String` - currently possible: \['update','skip','skip-minimizing-merge'\] - default: `'update'`                          |
 | `response-required`             | Configures for a **command** whether or not a **response** should be sent back.                                                              | `Boolean` - default: `true`                                                                                               |
 | `requested-acks`                | Defines which [acknowledgements](basic-acknowledgements.html) are requested for a command processed by Ditto.                                | `JsonArray` of `String` - default: `["twin-persisted"]`                                                                   |
 | `ditto-weak-ack`                | Marks [weak acknowledgements](basic-acknowledgements.html) issued by Ditto.                                                                  | `Boolean` - default: `false`                                                                                              |

documentation/src/main/resources/pages/ditto/release_notes_340.md

@@ -23,6 +23,7 @@ Eclipse Ditto 3.4.0 focuses on the following areas:
 * Addition of a **new placeholder** to use **in connections** to use **payload of the thing JSON** e.g. in headers or addresses
 * New **placeholder functions** for **joining** multiple elements into a single string and doing **URL-encoding and -decoding**
 * Configure **MQTT message expiry interval for published messages** via a header
+* **Reduce patch/merge thing commands** to **modify** only the **actually changed values** with a new option
 
 The following non-functional work is also included:
 
@@ -83,6 +84,17 @@ header `mqtt.message-expiry-interval` as part of a [MQTT5 target header mapping]
 dynamically influencing the MQTT message expiry interval, e.g. as part of a payload mapper for certain to-be-published 
 messages, or as a header mapping for all published messages.
 
+#### Reduce patch/merge thing commands to modify only the actually changed values with a new option
+
+In [#1772](https://github.com/eclipse-ditto/ditto/pull/1772) the existing [if-equal header](httpapi-concepts.html#conditional-headers)
+has been enhanced with a new option: `skip-minimizing-merge`.  
+Performing a [merge/patch command](protocol-specification-things-merge.html) and specifying this option as header will
+cause that the merge command's payload will be minimized to only the values which will actually be changed in the thing.
+
+This reduces e.g. required storage in the MongoDB a lot, if redundant data is often sent and
+also reduces the emitted event payload to [subscribers](basic-changenotifications.html) to only the actually changed 
+parts of the thing, reducing network load and making it more clear what actually changed with a "merge event".
+
 
 ### Changes
 

internal/utils/conditional-headers/src/main/java/org/eclipse/ditto/internal/utils/headers/conditional/IfEqualPreconditionHeader.java

@@ -246,7 +246,7 @@ private C adjustMergeCommandByOnlyKeepingChanges(final C command,
                     if (null == oldValue) {
                         return command;
                     } else if (command instanceof WithOptionalEntity<?> commandWithEntity) {
-                        return JsonMergePatch.compute(oldValue, newValue)
+                        return JsonMergePatch.compute(oldValue, newValue, false)
                                 .map(jsonMergePatch -> {
                                     final JsonValue jsonValue = jsonMergePatch.asJsonValue();
                                     return (C) commandWithEntity.setEntity(jsonValue);

json/src/main/java/org/eclipse/ditto/json/JsonMergePatch.java

@@ -43,26 +43,47 @@ private JsonMergePatch(final JsonValue mergePatch) {
      *
      * @param oldValue the original value
      * @param newValue the new changed value
-     * @return a JSON merge patch according to <a href="https://datatracker.ietf.org/doc/html/rfc7386">RFC 7386</a> or empty if values are equal.
+     * @return a JSON merge patch according to <a href="https://datatracker.ietf.org/doc/html/rfc7386">RFC 7386</a>
+     * or empty if values are equal.
      */
     public static Optional<JsonMergePatch> compute(final JsonValue oldValue, final JsonValue newValue) {
-        return computeForValue(oldValue, newValue).map(JsonMergePatch::of);
+        return compute(oldValue, newValue, true);
     }
 
-    private static Optional<JsonValue> computeForValue(final JsonValue oldValue, final JsonValue newValue) {
+    /**
+     * This method computes the change from the given {@code oldValue} to the given {@code newValue}.
+     * The result is a JSON merge patch according to <a href="https://datatracker.ietf.org/doc/html/rfc7386">RFC 7386</a>.
+     *
+     * @param oldValue the original value
+     * @param newValue the new changed value
+     * @param deleteMissingFieldsWithNull whether to delete fields contained in {@code oldValue} by setting those to
+     * {@code null} as defined in RFC 7386. This is the default behavior in order to create a valid JSON merge patch.
+     * @return a JSON merge patch according to <a href="https://datatracker.ietf.org/doc/html/rfc7386">RFC 7386</a>
+     * or empty if values are equal.
+     * @since 3.4.0
+     */
+    public static Optional<JsonMergePatch> compute(final JsonValue oldValue, final JsonValue newValue,
+            final boolean deleteMissingFieldsWithNull) {
+        return computeForValue(oldValue, newValue, deleteMissingFieldsWithNull).map(JsonMergePatch::of);
+    }
+
+    private static Optional<JsonValue> computeForValue(final JsonValue oldValue, final JsonValue newValue,
+            final boolean deleteMissingFieldsWithNull) {
         @Nullable final JsonValue diff;
         if (oldValue.equals(newValue)) {
             diff = null;
         } else if (oldValue.isObject() && newValue.isObject()) {
-            diff = computeForObject(oldValue.asObject(), newValue.asObject()).orElse(null);
+            diff = computeForObject(oldValue.asObject(), newValue.asObject(), deleteMissingFieldsWithNull)
+                    .orElse(null);
         } else {
             diff = newValue;
         }
         return Optional.ofNullable(diff);
     }
 
     private static Optional<JsonObject> computeForObject(final JsonObject oldJsonObject,
-            final JsonObject newJsonObject) {
+            final JsonObject newJsonObject,
+            final boolean deleteMissingFieldsWithNull) {
         final JsonObjectBuilder builder = JsonObject.newBuilder();
         final List<JsonKey> oldKeys = oldJsonObject.getKeys();
         final List<JsonKey> newKeys = newJsonObject.getKeys();
@@ -72,10 +93,12 @@ private static Optional<JsonObject> computeForObject(final JsonObject oldJsonObj
                 .collect(Collectors.toList());
         addedKeys.forEach(key -> newJsonObject.getValue(key).ifPresent(value -> builder.set(key, value)));
 
-        final List<JsonKey> deletedKeys = oldKeys.stream()
-                .filter(key -> !newKeys.contains(key))
-                .collect(Collectors.toList());
-        deletedKeys.forEach(key -> builder.set(key, JsonValue.nullLiteral()));
+        if (deleteMissingFieldsWithNull) {
+            final List<JsonKey> deletedKeys = oldKeys.stream()
+                    .filter(key -> !newKeys.contains(key))
+                    .collect(Collectors.toList());
+            deletedKeys.forEach(key -> builder.set(key, JsonValue.nullLiteral()));
+        }
 
         final List<JsonKey> keptKeys = oldKeys.stream()
                 .filter(newKeys::contains)
@@ -84,7 +107,8 @@ private static Optional<JsonObject> computeForObject(final JsonObject oldJsonObj
             final Optional<JsonValue> oldValue = oldJsonObject.getValue(key);
             final Optional<JsonValue> newValue = newJsonObject.getValue(key);
             if (oldValue.isPresent() && newValue.isPresent()) {
-                computeForValue(oldValue.get(), newValue.get()).ifPresent(diff -> builder.set(key, diff));
+                computeForValue(oldValue.get(), newValue.get(), deleteMissingFieldsWithNull)
+                        .ifPresent(diff -> builder.set(key, diff));
             } else if (oldValue.isPresent()) {
                 // Should never happen because deleted keys were handled before
                 builder.set(key, JsonValue.nullLiteral());
@@ -98,7 +122,7 @@ private static Optional<JsonObject> computeForObject(final JsonObject oldJsonObj
     }
 
     /**
-     * Creates a {@link JsonMergePatch} with an patch object containing the given {@code mergePatch} at the given {@code path}.
+     * Creates a {@link JsonMergePatch} with a patch object containing the given {@code mergePatch} at the given {@code path}.
      *
      * @param path The path on which the given {@code mergePatch} should be applied later.
      * @param mergePatch the actual patch.